Keras 实现类似连体的网络共享层

Question

我尝试在这里实现网络架构：Architecture of the network I try to implement (from Zhenyu et.al. 2019) using Keras framework with Tensorflow backend。

该网络是一个类似连体的网络，具有共享层 Conv1、Conv2、Conv3。目的是在 2 个不同的输入之间调整域，但这目前不是问题，我在此之前被卡住了。问题是我面临无法实现模型的问题。

所以我目前定义的模型如下：

为两个输入（源、目标）定义输入层：

source_domain_input = keras.layers.Input(shape=(None, None, 160, 1),name="SourceDomainInput")
target_domain_input = keras.layers.Input(shape=(None, None, 160, 1),name="TargetDomainInput")

然后我定义了共享层部分（Conv1、Conv2 和 Conv3）——为了缩短示例，我删除了池化层：

model_shared_part = keras.models.Sequential()
model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=16,
                                                                       kernel_size=3,
                                                                       activation='relu'),
                                                   input_shape=(None, None, 160, 1)))

model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=32,
                                                                       kernel_size=3,
                                                                       activation='relu')))

model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=64,
                                                                       kernel_size=3,
                                                                       activation='relu')))

然后我将输入连接到共享层以成为连体（并行）结构：

featureOutput_conv3_target = model_shared_part(target_domain_input)
featureOutput_conv3_source = model_shared_part(source_domain_input)

然后我将剩余的层添加到源部分（图像中的上部分支）以缩短示例我只添加了一些层而不是全部：

model_source_single_part = keras.models.Sequential()

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=64,
                                                                              kernel_size=3,
                                                                              activation='relu'),
                                                         )
                            )

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.MaxPooling1D(pool_size=2)
                                                          )
                             )

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.Flatten()
                                                         )
                            )

model_source_single_part.add(keras.layers.LSTM(units=60)
                            )

model_source_single_part.add(keras.layers.Dense(100,
                                                activation='relu')
                             )

model_source_single_part.add(keras.layers.Dense(units=4,
                                                activation='softmax'
                                               )
                            )

final_source_layer = model_source_single_part(featureOutput_conv3_source)

然后我将创建的架构合并到最终模型中：

dwcDAN_model = keras.models.Model(inputs=[source_domain_input,
                                          target_domain_input],
                                  outputs = final_source_layer
                                  )

# Model compilation
dwcDAN_model.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy'])

但是模型图给出了以下内容：

Model Plot

我所期望的是，名为sequential_2 的顺序模型是共享层部分，它连接到两个输入。但它只连接到源输入。

如何在上面的实现中找到错误？

Answer 1

想法很简单。连接 featureOutput_conv3_source 和 featureOutput_conv3_target 并将此连接功能与您的 final_source_layer 一起用作输出。现在您可能会问连接层的标签是什么。好吧，只需发送任何虚拟值，因为它永远不会被使用。我在下面给出一个大纲。请检查mmd损失函数中连接层的形状。

def compute_kernel(x, y):
    x_size = tf.shape(x)[0]
    print(x_size.shape)
    y_size = tf.shape(y)[0]
    print(y_size.shape)
    dim = tf.shape(x)[1]*tf.shape(x)[2]*tf.shape(x)[3]
    print(dim.shape)
    tiled_x = tf.tile(tf.reshape(x, tf.stack([x_size, 1, dim])), tf.stack([1, y_size, 1]))
    print(tiled_x.shape)
    tiled_y = tf.tile(tf.reshape(y, tf.stack([1, y_size, dim])), tf.stack([x_size, 1, 1]))
    print(tiled_y.shape)
    ret =  tf.exp(-tf.reduce_mean(tf.square(tiled_x - tiled_y), axis=2) / tf.cast(dim, tf.float32))
    print(ret.shape)
    return ret


def mmd(y_true, y_pred):
    print(y_pred.shape)
    #FEATURE_LENGTH = int(y_pred.shape[3]/2) # length of each is 64
    x1 = y_pred[:,:,:,:64]  # x1 represents feature output of conv3 from source
    x2 = y_pred[:,:,:,64:] # x2 represents feature output of conv3 from source
    x1x1 = compute_kernel(x1, x1)
    x1x2 = compute_kernel(x1, x2)
    x2x2 = compute_kernel(x2, x2)
    diff = keras.backend.mean(x1x1) - 2 * keras.backend.mean(x1x2) + keras.backend.mean(x2x2)
    return diff


model_shared_part = keras.models.Sequential()
model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=16,
                                                                       kernel_size=3,
                                                                       activation='relu'),
                                                   input_shape=(5, 160, 1)))

model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=32,
                                                                       kernel_size=3,
                                                                       activation='relu')))

model_shared_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=64,
                                                                       kernel_size=3,
                                                                       activation='relu')))


source_domain_input = keras.layers.Input(shape=(5, 160, 1),name="SourceDomainInput")
target_domain_input = keras.layers.Input(shape=(5, 160, 1),name="TargetDomainInput")
featureOutput_conv3_target = model_shared_part(target_domain_input)
featureOutput_conv3_source = model_shared_part(source_domain_input)


feature_concat = keras.layers.Concatenate()([featureOutput_conv3_source,featureOutput_conv3_target])
print(feature_concat.shape)



model_source_single_part = keras.models.Sequential()

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.Conv1D(filters=64,
                                                                              kernel_size=3,
                                                                              activation='relu'),
                                                         )
                            )

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.MaxPooling1D(pool_size=2)
                                                          )
                             )

model_source_single_part.add(keras.layers.TimeDistributed(keras.layers.Flatten()
                                                         )
                            )

model_source_single_part.add(keras.layers.LSTM(units=60)
                            )

model_source_single_part.add(keras.layers.Dense(100,
                                                activation='relu')
                             )

model_source_single_part.add(keras.layers.Dense(units=4,
                                                activation='softmax'
                                               )
                            )

final_source_layer = model_source_single_part(featureOutput_conv3_source)

print(final_source_layer.shape)


# Model build
dwcDAN_model = keras.models.Model(inputs=[source_domain_input, target_domain_input], outputs = [final_source_layer, feature_concat])
dwcDAN_model.compile(optimizer='adam', loss=['categorical_crossentropy', mmd])

print(dwcDAN_model.summary())


x1_train = np.random.rand(100, 5, 160, 1)
x2_train = np.random.rand(100, 5, 160, 1)
y_train1 = np.eye(100, 4)
y_train2 = np.random.rand(100, 5, 154, 128)


dwcDAN_model.fit([x1_train, x2_train], [y_train1, y_train2], epochs=1, batch_size=2, validation_split=.2)