TensorFlow 2.0 数据增强：tf.keras.preprocessing.image.ImageDataGenerator flow() 方法答案

【问题标题】：TensorFlow 2.0 Data Augmentation: tf.keras.preprocessing.image.ImageDataGenerator flow() methodTensorFlow 2.0 数据增强：tf.keras.preprocessing.image.ImageDataGenerator flow() 方法
【发布时间】：2020-10-14 14:39:00
【问题描述】：

我正在尝试使用 TensorFlow 2.2.0 和 Python 3.7 为 MNIST 数据集的 LeNet-300-100 密集神经网络执行数据增强。我的代码如下：

batch_size = 60
num_classes = 10
num_epochs = 100


# Data preprocessing and cleadning:
# input image dimensions
img_rows, img_cols = 28, 28

# Load MNIST dataset-
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()


if tf.keras.backend.image_data_format() == 'channels_first':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

print("\n'input_shape' which will be used = {0}\n".format(input_shape))
# 'input_shape' which will be used = (28, 28, 1)


# Convert datasets to floating point types-
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# Normalize the training and testing datasets-
X_train /= 255.0
X_test /= 255.0

# convert class vectors/target to binary class matrices or one-hot encoded values-
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)


X_train.shape, y_train.shape
# ((60000, 28, 28, 1), (60000, 10))

X_test.shape, y_test.shape
# ((10000, 28, 28, 1), (10000, 10))


# Example of using 'tf.keras.preprocessing.image.ImageDataGenerator class's - flow(x, y)':

datagen = ImageDataGenerator(
    # featurewise_center=True,
    # featurewise_std_normalization=True,
    rotation_range = 20,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    horizontal_flip = True
    )

现在，当我用代码看到“datagen.flow()”产生的批次数时：

# Sanity check-
i = 0

for x, y in datagen.flow(X_train, y_train, batch_size = batch_size, shuffle = True):
    # print("\ntype(x) = {0}, type(y) = {1}".format(type(x), type(y)))
    # print("x.shape = {0}, y.shape = {1}\n".format(x.shape, y.shape))
    print(i, end = ', ')
    i += 1

i 的值不断增加而不会终止。当然，出了点问题。据我所知，batch数=训练样例数/batch size。因此，在本例中，批次数 = 60000 / 60 = 1000。

那它为什么会产生这么多批次的增强数据呢？我该如何阻止它？怎么了？

谢谢！

【问题讨论】：

标签： python-3.x neural-network tensorflow2.0

【解决方案1】：

默认情况下，ImageDataGenerator 会无限生成图像。您可以像这里引用的那样在 for 循环中中断： How to find how many Image Generated By ImageDataGenerator

或者，您可以在学习时为 fit() 函数指定 steps_per_epoch 参数。

但是，tensorflow>=2.0 不支持多处理，所以 ImageDataGenerator 可能是学习的瓶颈。

【讨论】：