TensorFlow、Keras：替换预训练模型中的激活层

Question

我正在尝试在预训练的 TF 模型 EfficientNetB0 中将 swish 激活替换为 relu 激活。 EfficientNetB0 在 Conv2D 和 Activation 层中使用 swish 激活。这个SO post 与我正在寻找的非常相似。我还发现 an answer 适用于没有跳过连接的模型。下面是代码：

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import ReLU

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for layer in tuple(model.layers):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation.
                # Do something
                layer.activation = ReLU() # This didn't work
            else:
                # activation layer
                # Do something
                layer = tf.keras.layers.Activation('relu', name=layer.name + "_relu") # This didn't work
    return model

# load pretrained efficientNet
model = tf.keras.applications.EfficientNetB0(
    include_top=True, weights='imagenet', input_tensor=None,
    input_shape=(224, 224, 3), pooling=None, classes=1000,
    classifier_activation='softmax')

# convert swish activation to relu activation
model = replace_swish_with_relu(model)
model.save("efficientNet-relu")

如何修改replace_swish_with_relu在传递的模型中用relu替换swish激活？

感谢您的任何指点/帮助。

Answer 1

layer.activation 指向tf.keras.activations.swish 函数地址。我们可以修改它指向tf.keras.activations.relu。下面是修改后的replace_swish_with_relu：

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for layer in tuple(model.layers):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation
                layer.activation = tf.keras.activations.relu
            else:
                # activation layer
                layer.activation = tf.keras.activations.relu
    return model

注意：如果您正在修改激活函数，则需要重新训练模型以使用新的激活函数。 Related.

Answer 2

试试这个：

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for i,layer in enumerate(tuple(model.layers)):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation.
                # Do something
                model.layers[i] = ReLU() # This didn't work
            else:
                # activation layer
                # Do something
                model.layers[i] = tf.keras.layers.Activation('relu', name=layer.name + "_relu") # This didn't work
    return model