将 TF 记录加载到 Keras答案

【问题标题】：Loading TF Records into Keras将 TF 记录加载到 Keras
【发布时间】：2020-10-10 00:16:39
【问题描述】：

我正在尝试将自定义 TFRecord 文件加载到我的 keras 模型中。我尝试按照本教程进行操作：https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36，但正在适应我的使用。

我的目标是让功能类似于 Keras 的 ImageDataGenerator。我无法使用该功能，因为我从生成器未抓取的图像中获取了特定的元数据。我没有在此处包含该元数据，因为我只需要首先运行基本网络即可。

我还希望能够将其应用于迁移学习应用程序。

我不断收到此错误：TypeError: Could not build a TypeSpec for None with type NoneType 我正在使用 TensorFlow 2.2

def _parse_function(serialized):
    features = \
    {
        'image': tf.io.FixedLenFeature([], tf.string),
        'label': tf.io.FixedLenFeature([], tf.int64),
        'shapex': tf.io.FixedLenFeature([], tf.int64),
        'shapey': tf.io.FixedLenFeature([], tf.int64),
    }
    parsed_example = tf.io.parse_single_example(serialized=serialized,
                                                features=features)
    shapex = tf.cast(parsed_example['shapex'], tf.int32)
    shapey = tf.cast(parsed_example['shapey'], tf.int32)
    image_shape = tf.stack([shapex, shapey, 3])
    image_raw = parsed_example['image']
    # Decode the raw bytes so it becomes a tensor with type.
    image = tf.io.decode_raw(image_raw, tf.uint8)
    image = tf.reshape(image, image_shape)
    # Get labels
    label = tf.cast(parsed_example['label'], tf.float32)
    return image, label

def imgs_inputs(type, perform_shuffle=False):
    records_dir = '/path/to/tfrecord/'
    record_paths = [os.path.join(records_dir,record_name) for record_name in os.listdir(records_dir)]
    full_dataset = tf.data.TFRecordDataset(filenames=record_paths)
    full_dataset = full_dataset.map(_parse_function, num_parallel_calls=16)

    dataset_length = (len(list(full_dataset))) #Gets length of datase

    iterator = tf.compat.v1.data.make_one_shot_iterator(databatch)
    image, label = iterator.get_next()
    #labels saved as values ex: [1,2,3], and are now converted to one hot encoded
    label = to_categorical(label)
    return image, label

image, label = imgs_inputs(type ='Train',perform_shuffle=True)

#Combine it with keras
# base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(200,200,3), dropout=.3)
model_input = Input(shape=[200,200,3])

#Build your network
model_output = Flatten(input_shape=(200, 200, 3))(model_input)
model_output = Dense(19, activation='relu')(model_output)

#Create your model
train_model = Model(inputs=model_input, outputs=model_output)

#Compile your model
optimizer = Adam(learning_rate=.001)
train_model.compile(optimizer=optimizer,loss='mean_squared_error',metrics=['accuracy'],target_tensors=[label])

#Train the model
train_model.fit(epochs=10,steps_per_epoch=2)

image 返回形状为 (100,200,200,3) 的数组，这是一组 100 张图像 label 返回 shape(100,19) 的数组，这是一批 100 个标签（有 19 个标签）

【问题讨论】：

标签： python tensorflow keras tfrecord

【解决方案1】：

与shapex 和shapey 相关的问题，但我不知道具体原因。我设置了shapex = 200 和shapey=200。然后我重写了模型以包含迁移学习。

base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(200,200,3), dropout=.3)
x = base_model.output
types = Dense(19,activation='softmax')(x)

model = Model(inputs=base_model.input,outputs=types)

model.compile(
    optimizer='adam',
    loss = 'sparse_categorical_crossentropy',
    metrics=['accuracy']
history = model.fit(get_batches(), steps_per_epoch=1000, epochs=10)

I found everything I needed on this Google Colab:
[https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/04_Keras_Flowers_transfer_learning_solution.ipynb#scrollTo=XLJNVGwHUDy1][1]


  [1]: https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/04_Keras_Flowers_transfer_learning_solution.ipynb#scrollTo=XLJNVGwHUDy1

【讨论】：