在 keras 自定义层中包含高级计算（类 scikit）

Question

通常我会在将数据输入模型进行分类之前对其进行预处理。

然而这是不可能的，因此要么进一步（以某种方式）增强模型的性能，要么直接在模型中包含有用的预处理步骤。

我该怎么做？到目前为止，我发现的最佳解决方案包括使用 Keras 后端重新实现我想要的功能。这远不是一个好的解决方案，因此我希望有人有一个想法，如何挽救这种情况。

以下是我发现有用的链接 + 我当前的代码。

有用的链接：

Keras Custom Layer with advanced calculations

How to Switch from Keras Tensortype to numpy array for a custom layer?

How to create a Keras Custom Layer using functions not included in the Backend, to perform tensor sampling?

到目前为止我的代码：

def freezeBaseModelLayers(baseModel):
    for layer in baseModel.layers:
        layer.trainable = False


def preprocess_input(x):
    # TODO: Not working, but intention should be clear
    numpy_array = tf.unstack(tf.unstack(tf.unstack(x, 224, 0), 224, 0), 1, 0)
    from skimage.feature import hog
    from skimage import data, exposure
    img_adapteq = exposure.equalize_adapthist(numpy_array, orientations=8, pixels_per_cell=(3, 3),
                                              cells_per_block=(1, 1), visualize=True, multichannel=False)
    [x1, x2, x3] = tf.constant(img_adapteq), tf.constant(img_adapteq), tf.constant(img_adapteq)
    img_conc = Concatenate([x1, x2, x3])
    return img_conc


def create(x):
    is_training = tf.get_variable('is_training', (), dtype=tf.bool, trainable=False)
    with tf.name_scope('pretrained'):
        # Add preprocess step here...
        input_layer = Lambda(preprocess_input(x), input_shape=(224, 224, 1), output_shape=(224, 224, 3))

        baseModel = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))    
        freezeBaseModelLayers(baseModel)
        layer = baseModel(input_layer)
        layer = GlobalMaxPooling2D()(layer)
        layer = Dense(1024, activation='relu')(layer)
        layer = Dense(2, activation=None)(layer)
        model = Model(input=input_layer.input, output=layer)
        output = model(x)
        return output




I would like to include prepocessing steps inside my model

The models I am working with are receiving noisy data. In order to enhance the performance of the models, I would like to do some preprocessing steps e.g. equalize_adapthist.

Answer 1

要使用Lambda 层执行此操作，您需要在纯tensorflow 中编写直方图均衡化。 实际上，在构建图形时，函数 (preprocess_input) 将在 tensorflow 占位符上被调用，在这个需要 numpy 数组的 skimage 函数的情况下，它将不起作用。

This question 展示了如何用纯tensorflow 编写它。为了冗余/易于阅读，在此处复制粘贴（我自己没有测试过，但问题中有一个测试可用）：

def tf_equalize_histogram(image):
    values_range = tf.constant([0., 255.], dtype = tf.float32)
    histogram = tf.histogram_fixed_width(tf.to_float(image), values_range, 256)
    cdf = tf.cumsum(histogram)
    cdf_min = cdf[tf.reduce_min(tf.where(tf.greater(cdf, 0)))]

    img_shape = tf.shape(image)
    pix_cnt = img_shape[-3] * img_shape[-2]
    px_map = tf.round(tf.to_float(cdf - cdf_min) * 255. / tf.to_float(pix_cnt - 1))
    px_map = tf.cast(px_map, tf.uint8)

    eq_hist = tf.expand_dims(tf.gather_nd(px_map, tf.cast(image, tf.int32)), 2)
    return eq_hist

顺便说一句，您应该将预处理步骤编写为（一旦您将预处理步骤设为纯 tensorflow）：

input_layer = Lambda(preprocess_input, input_shape=(224, 224, 1), output_shape=(224, 224, 3))(x)

另一种方法是编写一个自定义层，正如 mlRocks 指出的那样。

Answer 2

更好的方法是通过自定义 keras 层。这是一个例子：

import tensorflow as tf
from keras.layers import Layer, Input, Conv2D
from keras.models import Model
from keras import backend as K
from skimage.feature import hog
from skimage import data, exposure

def equalize(img):
  img_adapteq = exposure.equalize_adapthist(img)
  return img_adapteq

def preprocess_input(img):
  return tf.py_func(equalize, 
                     [img],
                     'float32',
                     stateful=False,
                     name='custom_image_op')


class CustomLayer(Layer):
  def __init__(self, output_dim, **kwargs):
    self.output_dim = output_dim
    self.trainable = False
    super(CustomLayer, self).__init__(**kwargs)

  def call(self, x):
    res = tf.map_fn(preprocess_input, x)
    res.set_shape([x.shape[0],
                   self.output_dim[1], 
                   self.output_dim[0],
                   x.shape[-1]])
    return res

output_dim = (224,224)
inputs = Input(shape=(224,224,3))
x = CustomLayer(output_dim)(inputs)
x = Conv2D(32, (3,3))(x)
x = Flatten()(x)
x = Dense(1)(x)

model = Model(inputs, x)
model.summary()

# test
sample = np.random.rand(4, 224,224,3).astype(np.float32)
y = np.random.randint(2, size=(4,))

model.compile("sgd", "mse")
model.fit(sample, y)