如何将多个输入（四个）合并到神经网络中进行二值图像分类？答案

【问题标题】：How to merge multiple inputs (four) into neural network for binary image classification?如何将多个输入（四个）合并到神经网络中进行二值图像分类？
【发布时间】：2020-12-15 01:35:40
【问题描述】：

我将如何构建一个具有四个输入的二元分类神经网络（即，我不想输入一个完整的图像，而是想输入四个相同大小的图像块/图块，标记为 1 类或 2 类)。

我当前的实现遵循二进制类的简单顺序模型，我想将其转换为上述方案。

train_generator = train_datagen.flow_from_directory(
        r'MY\\TRAINING\\PATH',
        classes = ['Class1', 'Class2'],
        target_size=(100, 100), 
        batch_size=16,
        shuffle=False,
        class_mode='binary')

validation_generator = validation_datagen.flow_from_directory(
        r'MY\\VALIDATION\\PATH',
        classes = ['Class1', 'Class2'],
        target_size=(100, 100), 
        batch_size=8,
        # Use binary labels
        shuffle=False,
        class_mode='binary')

model = tf.keras.models.Sequential([
                                    tf.keras.layers.Conv2D(8, (3, 3), activation='relu', input_shape=(100, 100, 3)),
                                    tf.keras.layers.MaxPooling2D((2, 2)),
                                    tf.keras.layers.Conv2D(16, (3, 3), activation='relu'),
                                    tf.keras.layers.MaxPooling2D((2, 2)),
                                    tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
                                    tf.keras.layers.Flatten(),
                                    tf.keras.layers.Dense(1, activation='sigmoid')
                                    ])

例如，我的数据的文件结构如下所示：

├── training
│   ├── Class1_Quadrant1
│   │   ├── IMG_1.jpg
│   │   ├── IMG_2.jpg
│   │   ├── IMG_3.jpg
│   │   ├── etc.jpg
│   ├── Class1_Quadrant2
│   │   ├── IMG_1.jpg
│   │   ├── IMG_2.jpg
│   │   ├── IMG_3.jpg
│   │   ├── etc.jpg
│   ├── Class1_Quadrant3
│   │   ├── IMG_1.jpg
│   │   ├── IMG_2.jpg
│   │   ├── IMG_3.jpg
│   │   ├── etc.jpg
│   ├── Class1_Quadrant4
│   │   ├── IMG_1.jpg
│   │   ├── IMG_2.jpg
│   │   ├── IMG_3.jpg
│   │   ├── etc.jpg

并且我的验证集具有完全相同的结构。

理想情况下，如果有帮助的话，类似于这张图片：

编辑：

# Input Model 1
input1 = Input(shape=(100, 100, 3))
conv11 = Conv2D(32, kernel_size = 4, activation = 'relu')(input1)
pool11 = MaxPooling2D(pool_size = (2, 2))(conv11)
conv12 = Conv2D(16, kernel_size = 4, activation = 'relu')(pool11)
pool12 = MaxPooling2D(pool_size = (2, 2))(conv12)
conv12 = Conv2D(8, kernel_size = 4, activation = 'relu')(pool11)
pool12 = MaxPooling2D(pool_size = (2, 2))(conv12)
flat1 = Flatten()(pool12)
dense1 = Dense(1)(flat1)

# ... same structure repeated up to Input 4


# Merge All Input Models
merge = concatenate([flat1, flat2, flat3, flat4])

# Dense layers
hidden1 = Dense(1, activation = 'relu')(merge)
hidden2 = Dense(1, activation = 'relu')(hidden1)
hidden3 = Dense(1, activation = 'relu')(hidden2)
hidden4 = Dense(1, activation = 'relu')(hidden3)
output = Dense(1, activation = 'sigmoid')(hidden4)
model = Model(inputs = [input1, input2, input3, input4], outputs = output)

# Lastly my data comes from 

train_datagen = ImageDataGenerator(1 / 255.0)
validation_datagen = ImageDataGenerator(1 / 255.0)

train_generator1 = train_datagen.flow_from_directory(
    r'training\\path\\Q1',  
    classes = ['D1', 'D2'],
    target_size = (100, 100),  
    batch_size = 16,
    shuffle = False,
    class_mode = 'binary'
    )

validation_generator1 = validation_datagen.flow_from_directory(
    r'validation\\path\\Q1',  
    classes = ['D1', 'D2'],
    target_size = (100, 100), 
    batch_size = 8,
    shuffle = False,
    class_mode = 'binary'
    )

# this continues on for the 4 training and 4 validation generators
# until I get the error thrown here

model.compile(
    optimizer = tensorflow.optimizers.Adam(learning_rate = 1e-4),
    loss = 'binary_crossentropy',
    metrics = ['accuracy']
    )

history = model.fit(
    [train_generator1, train_generator2, train_generator3, train_generator4],  
    epochs = 100,
    verbose = 1,
    validation_data = [validation_generator1, validation_generator2, validation_generator3, validation_generator4],
    )

追溯：

history = model.fit(
    [train_generator1, train_generator2, train_generator3, train_generator4],  
    epochs = 100,
    verbose = 1,
    validation_data = [validation_generator1, validation_generator2, validation_generator3, validation_generator4],
    )
Traceback (most recent call last):

  File "<ipython-input-23-332d3f7e4bba>", line 5, in <module>
    validation_data = [validation_generator1, validation_generator2, validation_generator3, validation_generator4],

  File "C:\Users\Eitan Flor\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)

  File "C:\Users\Eitan Flor\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1063, in fit
    steps_per_execution=self._steps_per_execution)

  File "C:\Users\Eitan Flor\anaconda3\lib\site-packages\tensorflow\python\keras\engine\data_adapter.py", line 1104, in __init__
    adapter_cls = select_data_adapter(x, y)

  File "C:\Users\Eitan Flor\anaconda3\lib\site-packages\tensorflow\python\keras\engine\data_adapter.py", line 971, in select_data_adapter
    _type_name(x), _type_name(y)))

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'tensorflow.python.keras.preprocessing.image.DirectoryIterator'>"}), <class 'NoneType'>

【问题讨论】：

嗨，你能解决这个问题吗？请让我知道它是如何为你工作的。谢谢

标签： python tensorflow keras conv-neural-network

【解决方案1】：

使用 Sequential API 是不可能的（每一层都应该有一个输入）：https://www.tensorflow.org/guide/keras/sequential_model

试用函数式 API：https://www.tensorflow.org/guide/keras/functional

【讨论】：

谢谢，我继续使用函数式 API 方法开始转换。我现在坚持训练模型，它抛出“ValueError：无法找到可以处理输入的数据适配器：（包含类型的值{“"}), "，我将在我的原始帖子中添加回溯和更新代码以供参考。
@EitanFlor 你如何生成标签？
这就是我使用数据生成器的原因，例如，如果我检查“train_generator1.labels”，它会生成一个标签数组，例如：array([0,0,0,.. .1,1,1,...]）。我最近尝试在 model.fit() 中输入图像的实际 numpy 数组（我将它加载到四个临时变量中，而不是直接传递 datagen 并解决了这个问题）。我几乎明白了，现在只有一个大小不匹配，我正在尝试解决： ValueError: Input 0 of layer dense_4 is incompatible with the layer: expected axis -1 of input shape to have value 15488 but received input with shape [None, 964800].
@EitanFlor 请查看stackoverflow.com/help/how-to-ask。发帖前需自行调查
@IvanAracki 感谢您的评论。在问题的上下文中 - 功能 API 的基本部分是它可以用于创建具有少量输入的模型。它包含在答案中。如您所见 - 它有帮助

【解决方案2】：

我怀疑您必须创建一个具有 4 个输入的模型，每个图像象限一个输入。然后每个输入可以根据需要由层独立处理。然后展平每个输入流的输出并将它们连接起来。然后使用具有 1 个节点的密集层处理连接层输出。下面的代码是关于如何生成输入的一种想法。有一个培训目录，其中包含作为类的子目录，并将图像放入这些子目录中。下面的示例展示了如何从 ImageDataGenerator 中获取图像并将图像分成象限。

# function below splits an image into 4 quadrants and returns them in imgq
def quad_split (img, width, height):
    mid_width=width//2
    mid_height=height//2
    imgq=[]
    imgq.append(img[0:mid_height, 0:mid_width])
    imgq.append(img[0:mid_height, mid_width:])
    imgq.append(img[mid_height:, 0:mid_width])
    imgq.append(img[mid_height:, mid_width:])
    return imgq
# function below just  displays the 4 quadrants 
def quad_plot(quad):
    fig = plt.figure(figsize=( 10, 10))
    for i in range (4):
        imgn=(quad[i]+1)/2
        fig.add_subplot(2, 2, i+1)
        plt.imshow(imgn)
train_dir=r'c:\temp\people\train' # point this to your training directory
train_gen=ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet.preprocess_input, horizontal_flip=True)
train_gen=train_gen.flow_from_directory( train_dir, target_size=(224, 224), batch_size=32, seed=123, )
# you can make similar generators for validation and test data
images, labels=next(train_gen) # fetch the next batch of images and labels
for image in images:
    quads=quad_split (image, 224, 224) # split each image into quadrants
    quad_plot(quads) # just used to display the quadrants of an image

您必须构建一个生成器，为模型创建 4 个输入，包括标签，但这应该不难实现。祝你好运

【讨论】：