将 Keras ConvLSTM2D 输出转换为输入到 Conv2D

Question

我有一个用于输入图像的模型，该模型的目的是与作为输出的单个图像进行比较。

所有/每个图像大小 = [96,96,3]

模型在一个文件夹中总共有 6 张图像，其中 5 张与第 6 张图像有点相似，我们必须使用 5 张图像（图像增强图像）来正确预测第 6 张图像. 测试集和训练集有多个这样的文件夹

我决定使用 CNN+LSTM 来解决这个问题，我相信它会比仅使用普通 CNN 来解决这个问题做得更好，我已经尝试了普通 CNN 和图像增强，它们的性能达到了顶峰。

以下是我目前拥有的模型，评论的部分是我遇到问题的地方。

config.height = 96
config.width = 96
model = Sequential()
model.add(ConvLSTM2D(filters=32, kernel_size=(3,3), padding='same', input_shape=(5, config.height, config.width, 3), return_sequences=True, stateful=False))
model.add(BatchNormalization())
model.add(ConvLSTM2D(filters=32, kernel_size=(3,3), padding='same', return_sequences=True))
model.add(BatchNormalization())
model.add(ConvLSTM2D(filters=32, kernel_size=(3,3), padding='same', return_sequences=True))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(ConvLSTM2D(filters=32, kernel_size=(3,3), padding='same', return_sequences=True))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(ConvLSTM2D(filters=3, kernel_size=(3,3), padding='same', return_sequences=True))
model.add(BatchNormalization())
# LSTM_to_conv_dims = (-1, 96, 96, 3)
# model.add(Reshape(LSTM_to_conv_dims))
print(f'Shape of model {model.summary()}')
model.add(Conv2D(3, (3, 3), activation='relu', padding='same'))

为了帮助这个模型的输出是上面的model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
conv_lst_m2d_1 (ConvLSTM2D)  (None, 5, 96, 96, 32)     40448     
_________________________________________________________________
batch_normalization_1 (Batch (None, 5, 96, 96, 32)     128       
_________________________________________________________________
conv_lst_m2d_2 (ConvLSTM2D)  (None, 5, 96, 96, 32)     73856     
_________________________________________________________________
batch_normalization_2 (Batch (None, 5, 96, 96, 32)     128       
_________________________________________________________________
conv_lst_m2d_3 (ConvLSTM2D)  (None, 5, 96, 96, 32)     73856     
_________________________________________________________________
batch_normalization_3 (Batch (None, 5, 96, 96, 32)     128       
_________________________________________________________________
dropout_1 (Dropout)          (None, 5, 96, 96, 32)     0         
_________________________________________________________________
conv_lst_m2d_4 (ConvLSTM2D)  (None, 5, 96, 96, 32)     73856     
_________________________________________________________________
batch_normalization_4 (Batch (None, 5, 96, 96, 32)     128       
_________________________________________________________________
dropout_2 (Dropout)          (None, 5, 96, 96, 32)     0         
_________________________________________________________________
conv_lst_m2d_5 (ConvLSTM2D)  (None, 5, 96, 96, 3)      3792      
_________________________________________________________________
batch_normalization_5 (Batch (None, 5, 96, 96, 3)      12        
_________________________________________________________________
reshape_1 (Reshape)          (None, 5, 96, 96, 3)      0         
=================================================================
Total params: 266,332
Trainable params: 266,070
Non-trainable params: 262
_________________________________________________________________

• 当我尝试 重塑并将我的 BatchNorm_5 输出作为输入输入到 Conv2D。

• 据我了解，需要将其更改为 [None, 96, 96, 3]。 这就是我需要社区帮助的地方。

• 为什么我需要这个形状，是因为我的输出图像是相同的
  需要比较的形状。

• 我想以最少的信息损失进行此重塑。

我得到的错误，非常正确，但我需要有关如何纠正此错误的想法

Traceback (most recent call last):
  File "train.py", line 154, in <module>
    model.add(Conv2D(3, (3, 3), activation='relu', padding='same'))
  File "/home/sandeeppanku/anaconda3/envs/tensorflowpy1p11/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
    output_tensor = layer(self.outputs[0])
  File "/home/sandeeppanku/anaconda3/envs/tensorflowpy1p11/lib/python3.6/site-packages/keras/engine/base_layer.py", line 414, in __call__
    self.assert_input_compatibility(inputs)
  File "/home/sandeeppanku/anaconda3/envs/tensorflowpy1p11/lib/python3.6/site-packages/keras/engine/base_layer.py", line 311, in assert_input_compatibility
    str(K.ndim(x)))
ValueError: Input 0 is incompatible with layer conv2d_1: expected ndim=4, found ndim=5

Answer 1

如果在最后一个 convlstm2d 层使用conv2d，则必须将return_sequences=True 更改为return_sequences=False。运行程序后，可以得到ndim=4。