用于视频输入的 LSTM

Question

我是一个尝试 LSTM 的新手。

我基本上使用 LSTM 来确定动作类型（5 种不同的动作），例如跑步、跳舞等。我的输入是每个动作 60 帧，大致可以说大约 120 个这样的视频

train_x.shape = (120,192,192,60)

其中 120 是用于训练的样本视频数，192X192 是帧大小，60 是 # 帧。

train_y.shape = (120*5) [1 0 0 0 0 ..... 0 0 0 0 1] 一个热编码

我不清楚如何将 3d 参数传递给 lstm（时间戳和特征）

model.add(LSTM(100, input_shape=(train_x.shape[1],train_x.shape[2])))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dense(len(uniquesegments), activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(train_x, train_y, epochs=100, batch_size=batch_size, verbose=1)

我收到以下错误

层序的输入 0 与层不兼容：预期 ndim=3，发现 ndim=4。收到的完整形状：（无、192、192、60）

训练数据算法

Loop through videos
            Loop through each frame of a video
                    logic
                    append to array
            convert to numpy array
            roll axis to convert 60 192 192 to 192 192 60
  add to training list
convert training list to numpy array

训练列表形状

Answer 1

首先你应该知道，解决视频分类任务的方法比LSTM或任何RNN Cell更适合卷积RNN，就像CNN 比 MLP

更适合图像分类任务

那些 RNN 单元（例如 LSTM、GRU）预计输入形状为 (samples, timesteps, channels)，因为您处理的输入形状为 (samples, timesteps, width, height, channels)，因此您应该改用 tf.keras.layers.ConvLSTM2D

以下示例代码将向您展示如何构建一个可以处理您的视频分类任务的模型：

import tensorflow as tf
from tensorflow.keras import models, layers

timesteps = 60
width = 192
height = 192
channels = 1
action_num = 5

model = models.Sequential(
    [
        layers.Input(
            shape=(timesteps, width, height, channels)
        ),
        layers.ConvLSTM2D(
            filters=64, kernel_size=(3, 3), padding="same", return_sequences=True, dropout=0.1, recurrent_dropout=0.1
        ),
        layers.MaxPool3D(
            pool_size=(1, 2, 2), strides=(1, 2, 2), padding="same"
        ),
        layers.BatchNormalization(),
        layers.ConvLSTM2D(
            filters=32, kernel_size=(3, 3), padding="same", return_sequences=True, dropout=0.1, recurrent_dropout=0.1
        ),
        layers.MaxPool3D(
            pool_size=(1, 2, 2), strides=(1, 2, 2), padding="same"
        ),
        layers.BatchNormalization(),
        layers.ConvLSTM2D(
            filters=16, kernel_size=(3, 3), padding="same", return_sequences=False, dropout=0.1, recurrent_dropout=0.1
        ),
        layers.MaxPool2D(
            pool_size=(2, 2), strides=(2, 2), padding="same"
        ),
        layers.BatchNormalization(),
        layers.Flatten(),
        layers.Dense(256, activation='relu'),
        layers.Dense(action_num, activation='softmax')
    ]
)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

输出：

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv_lst_m2d (ConvLSTM2D)    (None, 60, 192, 192, 64)  150016    
_________________________________________________________________
max_pooling3d (MaxPooling3D) (None, 60, 96, 96, 64)    0         
_________________________________________________________________
batch_normalization (BatchNo (None, 60, 96, 96, 64)    256       
_________________________________________________________________
conv_lst_m2d_1 (ConvLSTM2D)  (None, 60, 96, 96, 32)    110720    
_________________________________________________________________
max_pooling3d_1 (MaxPooling3 (None, 60, 48, 48, 32)    0         
_________________________________________________________________
batch_normalization_1 (Batch (None, 60, 48, 48, 32)    128       
_________________________________________________________________
conv_lst_m2d_2 (ConvLSTM2D)  (None, 48, 48, 16)        27712     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 24, 24, 16)        0         
_________________________________________________________________
batch_normalization_2 (Batch (None, 24, 24, 16)        64        
_________________________________________________________________
flatten (Flatten)            (None, 9216)              0         
_________________________________________________________________
dense (Dense)                (None, 256)               2359552   
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 1285      
=================================================================
Total params: 2,649,733
Trainable params: 2,649,509
Non-trainable params: 224
_________________________________________________________________

请注意，在输入上述模型之前，您应该将数据重新排序为形状(samples, timesteps, width, height, channels)（即不像np.reshape，而是像np.moveaxis），在您的情况下，形状应该是(120, 60, 192, 192, 1)，然后您可以拆分你的 120 视频批量和饲料模型

Answer 2

从文档看来，LSTM 似乎甚至不打算采用 input_shape 参数。这是有道理的，因为通常您应该为每个时间步提供一维特征。这就是为什么在文档中它说：

输入：具有形状 [batch, timesteps, feature] 的 3D 张量

您尝试做的事情不会奏效（我还给您留下了评论，解释了为什么您可能不应该尝试这样做）。