Keras 混合功能模型中的图像增强

Question

我在Keras 中创建了一个混合模型，为元数据和图像数据创建权重，然后将它们组合起来进行分类。这是模型：

Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 80, 120, 3)] 0                                            
__________________________________________________________________________________________________
xception (Functional)           (None, 3, 4, 2048)   20861480    input_5[0][0]                    
__________________________________________________________________________________________________
input_4 (InputLayer)            [(None, 10)]         0                                            
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 3, 4, 8)      409608      xception[0][0]                   
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 4)            44          input_4[0][0]                    
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8)            0           conv2d_9[0][0]                   
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 12)           0           dense_3[0][0]                    
                                                                 global_average_pooling2d_1[0][0] 
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 4)            52          concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 1)            5           dense_4[0][0]                    
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________

由于不平衡，我决定增加图像。我使用了以下 ImageDataGenerator：

aug = ImageDataGenerator(rescale=1/255.,
                        rotation_range=180,
                        height_shift_range=0.2,
                        width_shift_range=0.2,
                        brightness_range=[0.5,1.5],
                        channel_shift_range=100.0,
                        horizontal_flip=True,
                        vertical_flip=True,
                        shear_range=45.0)

然后我编译并尝试使用ImageDataGenerator().flow() 训练模型：

epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)

history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)

这给了我一个错误：

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>

我尝试了多个版本的代码，但我对后端不够熟悉，无法正确诊断问题。谁能帮我解决这个问题？

---

型号代码和MRE

型号代码

LEARNING_RATE = 0.001

# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))

# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)

# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)

# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)


# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)

# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()

meta_train MRE

       age_approx  Unknown  female  male  head/neck  lower extremity  \
11655          45        0       0     1          0                0   
24502          60        0       0     1          0                1   
2524           50        0       1     0          0                1   
13894          60        0       1     0          0                0   
29325          45        0       1     0          0                1   

       oral/genital  palms/soles  torso  upper extremity  
11655             0            0      1                0  
24502             0            0      0                0  
2524              0            0      0                0  
13894             0            0      1                0  
29325             0            0      0                0 

img_train MRE

数组太大，见代码here。

y_train.shape

(23188, 1)

Answer 1

首先，如果您生成以下数组，您的模型可以正常工作；

import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001

# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))

# Model 1
meta_layer1 = tf.keras.layers.Dense(4, activation='relu')(meta_inputs)

# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)

# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4, activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1, activation='sigmoid')(merged_dense_layer)


# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)

# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()

让我们生成如下数组；

y_array = np.zeros((23188, 1))
train_array = np.zeros((23188, 80, 120,3))
meta_array = np.zeros((23188, 10))

现在测试你的模型；

model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)

你可以看到它运行

>>> model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
 47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00

现在问题出在您的输入中：

(1) 首先将所有元数据转换为一个数组，假设它有 1000 个批次，因此它的形状必须为 meta_train.shape 是 (1000,10)

(2) 你的img_train.shape 是(1000,80,120,3)

(3) 你的y_train.shape 是(1000,1)

这里的 1000 也可以是 23188。但是让我们假设您有 1000 张图像、y_train（目标）和 1000 个元数据。

当你想增加你的图像训练时，你必须在这里小心。如下使用;

import tensorflow as tf
import numpy as np
import time

在这里，我创建了空数组作为示例，但您的原始数据必须以如下形状的形式排列在数组中，除了 1000 之外，它必须是您拥有的图像数量。

y_train = np.zeros((1000, 1))
img_train = np.zeros((1000, 80, 120,3))
meta_train= np.zeros((1000, 10))

aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,
                        rotation_range=180,
                        height_shift_range=0.2,
                        width_shift_range=0.2,
                        brightness_range=[0.5,1.5],
                        channel_shift_range=100.0,
                        horizontal_flip=True,
                        vertical_flip=True,
                        shear_range=45.0)

aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false, you have to concatenate at each loop the meta_train and y_train for each image, so do not shuffle the images!
imagenerator = aug.flow(img_train, batch_size = img_train.shape[0], shuffle=False, sample_weight=None, seed=123, save_to_dir=None, save_prefix="", save_format="png", subset=None)

new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy()       # concatenate y_train at each loop to new_y


batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
    print(batches, time.strftime("%Y:%m%d-%H:%M:%S"))
    batches += 1
    img_train = np.concatenate((img_train, x_batch), axis = 0)
    new_meta = np.concatenate((new_meta, meta_train), axis = 0) # concatenate corresponding meta data
    new_y = np.concatenate((new_y, y_train), axis = 0) #concatenate corresponding label/target data!
    if batches >= 5:
        break

model.fit(x = [new_meta, img_train], y = new_y, epochs = 1, batch_size = 32)

注意：在model.fit 中使用它们之前，您可以像下面这样简单地打乱您的数据

idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)

img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]

Answer 2

这不是直接的答案，但有助于解决问题。缺少解决您问题的信息。

但首先：最明显的问题是；你提供meta_train 这是一个熊猫数据框。将其转换为数组。 请先试试这个。

第二，如果你仍然有问题，那么model.fit 可能无法处理列表[meta_train, flow]，那么你可能需要找到一种方法来提供两个输入供模型处理。

为此，您应该以MRE 的形式提供以下信息，以便重现您的问题。

(1) 您的模型代码，即使根据提供的摘要可能会生成相同的模型。

(2) y_train 的形状。