使用 fit_generator 的 Keras 迁移学习 Resnet50 出现高 acc 但低 val_acc 问题

Question

我正在使用 Resnet50 模型进行迁移学习，使用 100,000 张图像，共 20 个场景（MIT Place365 数据集）。我只训练了最后 160 层（由于内存限制）。问题是我得到了相当高的准确度但验证准确度极低，我认为这可能是一个过度拟合的问题，但我不知道如何解决它。如果有人能给我关于如何解决我的低 val_acc 问题的建议，我将非常感激，非常感谢。 我的代码如下：

V1 = np.load("C:/Users/Desktop/numpydataKeras_20_val/imgonehot_val_500.npy")
V2 = np.load("C:/Users/Desktop/numpydataKeras_20_val/labelonehot_val_500.npy") 


net = keras.applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=None, input_shape=(224, 224, 3))

x = net.output
x = Flatten()(x)
x = Dense(128)(x)
x = Activation('relu')(x)
x = Dropout(0.5)(x)
output_layer = Dense(20, activation='softmax', name='softmax')(x)
net_final = Model(inputs=net.input, outputs=output_layer)

for layer in net_final.layers[:-160]:
    layer.trainable = False
for layer in net_final.layers[-160:]:
    layer.trainable = True

net_final.compile(Adam(lr=.00002122), loss='categorical_crossentropy', metrics=['accuracy'])

def data_generator():
    n = 100000
    Num_batch = 100000/100
    arr = np.arange(1000)
    np.random.shuffle(arr)
    while (True):
        for i in arr:
            seed01 = random.randint(0,1000000)

            X_batch  = np.load( "C:/Users/Desktop/numpydataKeras/imgonehot_"+str((i+1)*100)+".npy" )
            np.random.seed(seed01)
            np.random.shuffle(X_batch)

            y_batch = np.load( "C:/Users/Desktop/numpydataKeras/labelonehot_"+str((i+1)*100)+".npy" )
            np.random.seed(seed01)
            np.random.shuffle(y_batch)

            yield X_batch, y_batch

weights_file = 'C:/Users/Desktop/Transfer_learning_resnet50_fit_generator_02s.h5'
early_stopping = EarlyStopping(monitor='val_acc', patience=5, mode='auto', verbose=2)
model_checkpoint = ModelCheckpoint(weights_file, monitor='val_acc', save_best_only=True, verbose=2)
callbacks = [early_stopping, model_checkpoint]

model_fit = net_final.fit_generator(
    data_generator(),
    steps_per_epoch=1000,
    epochs=5,
    validation_data=(V1, V2),
    callbacks=callbacks,
    verbose=1,
    pickle_safe=False)

以下是打印输出：

Epoch 1/5
1000/1000 [==============================] - 3481s 3s/step - loss: 1.7917 - acc: 0.4757 - val_loss: 3.5872 - val_acc: 0.0560

Epoch 00001: val_acc improved from -inf to 0.05600, saving model to C:/Users/Desktop/Transfer_learning_resnet50_fit_generator_02s.h5
Epoch 2/5
1000/1000 [==============================] - 4884s 5s/step - loss: 1.1287 - acc: 0.6595 - val_loss: 4.2113 - val_acc: 0.0520

Epoch 00002: val_acc did not improve from 0.05600
Epoch 3/5
1000/1000 [==============================] - 4964s 5s/step - loss: 0.8033 - acc: 0.7464 - val_loss: 4.9595 - val_acc: 0.0520

Epoch 00003: val_acc did not improve from 0.05600
Epoch 4/5
1000/1000 [==============================] - 4961s 5s/step - loss: 0.5677 - acc: 0.8143 - val_loss: 4.5484 - val_acc: 0.0520

Epoch 00004: val_acc did not improve from 0.05600
Epoch 5/5
1000/1000 [==============================] - 4928s 5s/step - loss: 0.3999 - acc: 0.8672 - val_loss: 4.6155 - val_acc: 0.0400

Epoch 00005: val_acc did not improve from 0.05600

Answer 1

按照https://github.com/keras-team/keras/issues/9214#issuecomment-397916155 看来，批量标准化应该是可训练的。

以下代码可以替换您设置/取消设置可训练层的循环：

for layer in model.layers:
    if hasattr(layer, 'moving_mean') and hasattr(layer, 'moving_variance'):
        layer.trainable = True
        K.eval(K.update(layer.moving_mean, K.zeros_like(layer.moving_mean)))
        K.eval(K.update(layer.moving_variance, K.zeros_like(layer.moving_variance)))
    else:
        layer.trainable = False

根据我自己的数据，我需要减少批量大小以避免 OOM，我现在有：

Epoch 1/10
470/470 [==============================] - 90s 192ms/step - loss: 0.3513 - acc: 0.8660 - val_loss: 0.1299 - val_acc: 0.9590
Epoch 2/10
470/470 [==============================] - 83s 177ms/step - loss: 0.2204 - acc: 0.9163 - val_loss: 0.1276 - val_acc: 0.9471
Epoch 3/10
470/470 [==============================] - 83s 177ms/step - loss: 0.2219 - acc: 0.9184 - val_loss: 0.1048 - val_acc: 0.9589
Epoch 4/10
470/470 [==============================] - 83s 177ms/step - loss: 0.1813 - acc: 0.9327 - val_loss: 0.1857 - val_acc: 0.9303

警告，它可能会影响准确性，您必须冻结模型以避免奇怪的推理。但这似乎是唯一对我有用的方法。

另一条评论https://github.com/keras-team/keras/issues/9214#issuecomment-422490253 仅检查层名称以将其设置为可训练，如果它是批量标准化，但它对我没有任何改变。也许它可以帮助您的数据集。