TensorFlow Data 不适用于多输入 keras 模型

Question

我正在使用 TensorFlow Data 和 keras 构建管道来训练我的模型，直到现在一切正常，使用单个输入图像。事实证明，我需要将 2 个具有单独 CNN 架构的新图像合并到单个神经网络中，总共 3 个输入。让我展示一下代码：

def read_tfrecord(example_proto):
    image_feature_description = {
    'label': tf.io.FixedLenFeature([], tf.int64),
    'full_image': tf.io.FixedLenFeature([], tf.string),
    'hand_0': tf.io.FixedLenFeature([], tf.string),
    'hand_1': tf.io.FixedLenFeature([], tf.string)
    }

    example = tf.io.parse_single_example(example_proto, image_feature_description)
    return read_full_image_and_label(example)

def load_dataset(tf_record_path):
    raw_dataset = tf.data.TFRecordDataset(tf_record_path, num_parallel_reads=AUTOTUNE)

    parsed_dataset = raw_dataset.map(read_tfrecord, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    return parsed_dataset 

def load_data_tfrecord(tfrecord_path, augmentation):
  dataset = load_dataset(tfrecord_path)

  dataset = prepare_for_training(dataset, augmentation)
  return dataset

data_generator['train'] = load_data_tfrecord(train_fns, True)

我按照这个顺序读取我的 TF 记录，直到此时代码几乎与之前的工作相同，只是添加了“hand_0”和“hand_1”（我的新图像）。 为了得到我遵循这种方法的图像：

def decode_and_resize_img(tf_btstring, img_width, img_height):
  image = tf.image.decode_jpeg(tf_btstring, channels=3)
  image = tf.image.resize(image, [img_width, img_height])

  return image 

def read_full_image_and_label(tf_bytestring):
    image = decode_and_resize_img(tf_bytestring['full_image'], 224, 224)
    hand_0 = decode_and_resize_img(tf_bytestring['hand_0'], 224, 224)
    hand_1 = decode_and_resize_img(tf_bytestring['hand_1'], 224, 224)  

    label = tf.cast(tf_bytestring['label'], tf.int32)

    return {'image_data': image, 'hnd_0': hand_0, 'hnd_1': hand_1}, label

我创建了一个对象来放置我的所有图像，而不是只返回一个图像和一个标签。在最后一步中，我准备数据集来为模型提供数据：

def prepare_for_training(ds, augmentation=True, shuffle_buffer_size=1000):
    ds.cache()
    ds = ds.repeat()
    ds = ds.batch(batch_size)

    ds = ds.unbatch()
    ds = ds.shuffle(buffer_size=shuffle_buffer_size)
    ds = ds.batch(batch_size)
    ds = ds.prefetch(AUTOTUNE)
    return ds

在整个架构中，我只是通过一个对象和 TFRecord 描述改变了图像的返回。我也改变了CNN/NN拓扑，如下：

    full_body_model = EfficientNetB4(input_shape=(img_width, img_height, 3), weights='imagenet', include_top=False)
    hand_1_model =  EfficientNetB0(input_shape=(224, 224, 3), weights='imagenet', include_top=False)
    hand_2_model =  EfficientNetB0(input_shape=(224, 224, 3), weights='imagenet', include_top=False)

    full_body_model.trainable = False
    hand_1_model.trainable = False
    hand_2_model.trainable = False

    for layer in hand_1_model.layers:
      layer._name = 'hand_1' + str(layer._name)

    for layer in hand_2_model.layers:
      layer._name = 'hand_2' + str(layer._name)

    fbm = full_body_model.output
    fbm = GlobalAveragePooling2D()(fbm)

    h1m = hand_1_model.output
    h1m = GlobalAveragePooling2D()(h1m)
    
    h2m = hand_2_model.output
    h2m = GlobalAveragePooling2D()(h2m)

    x = Concatenate()([fbm, h1m, h2m])
    x = Dense(512, activation='elu', name='fc1')(x)
    x = Dropout(0.4)(x)
    x = Dense(256, activation='elu', name='fc2')(x)
    x = Dropout(0.4)(x)
    x = Dense(128, activation='elu', name='fc3')(x)
    x = Dropout(0.4)(x)
    x = Dense(64, activation='elu', name='fc4')(x)
    x = Dropout(0.4)(x)

    predictions = Dense(nb_classes, activation='softmax', name='predictions')(x)

    model = Model([full_body_model.input, hand_1_model.input, hand_2_model.input], predictions)

现在，我遇到了这个错误：

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:811 train_function  *
        for _ in math_ops.range(self._steps_per_execution):
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:414 for_stmt
        symbol_names, opts)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:629 _tf_range_for_stmt
        opts)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:1059 _tf_while_stmt
        body, get_state, set_state, init_vars, nulls, symbol_names)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:1032 _try_handling_undefineds
        _verify_loop_init_vars(init_vars, symbol_names, first_iter_vars)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:193 _verify_loop_init_vars
        raise ValueError(error_msg)

    ValueError: 'outputs' must be defined before the loop.

我尝试过其他方法，比如这里：https://github.com/tensorflow/tensorflow/issues/20698，但我总是遇到同样的错误。我很想得到一些帮助来解决这个问题！

我使用的是当前版本的 Google Colab，带有 TF 2.4.0

Answer 1

为了解决这个问题，我更改了模型接收输入数据的方式。我正在使用自定义生成器将多输入图像堆叠在一起并将此列表传递给 fit 方法。这是我的代码：

    def train_gen():
      for (stacked, label) in train_ds:
        label = label 
        img1 = stacked[:,0,:,:]    
        img2 = stacked[:,1,:,:]
        img3 = stacked[:,2,:,:]
        img1, img2, img3 = transform_batch(img1, img2, img3) #data augmentation
        yield [img1, img2, img3], label