如何将 mnist 数据集拆分成更小的尺寸并对其进行扩充？答案

【问题标题】：How to split mnist dataset into smaller size and adding augmentation to it?如何将 mnist 数据集拆分成更小的尺寸并对其进行扩充？
【发布时间】：2023-01-20 21:20:52
【问题描述】：

我有拆分 mnist 数据集 + 添加扩充数据的问题。我只想从 mnist 数据集中获取 22000（包括训练 + 测试集）数据，即 70000。mnist 数据集有 10 个标签。我只使用剪切、旋转、宽度偏移和高度偏移来进行增强方法。

训练集 --> 20000（总计）--> 20 张图像 + 1980 张增强图像（每个标签）

测试集 --> 2000（总计）--> 200 张图片（每个标签）

我还想确保在拆分中保留类分布。

我真的很困惑如何拆分这些数据。如果有人能提供代码，我会很高兴。

我试过这段代码：

# Load the MNIST dataset
(x_train_full, y_train_full), (x_test_full, y_test_full) = keras.datasets.mnist.load_data()

# Normalize the data
x_train_full = x_train_full / 255.0
x_test_full = x_test_full / 255.0

# Create a data generator for data augmentation
data_gen = ImageDataGenerator(shear_range=0.2, rotation_range=20,
                              width_shift_range=0.2, height_shift_range=0.2)

# Initialize empty lists for the training and test sets
x_train, y_train, x_test, y_test = [], [], [], []

# Loop through each class/label
for class_n in range(10):
    # Get the indices of the images for this class
    class_indices = np.where(y_train_full == class_n)[0]

    # Select 20 images for training
    train_indices = np.random.choice(class_indices, 20, replace=False)

    # Append the training images and labels to the respective lists
    x_train.append(x_train_full[train_indices])
    y_train.append(y_train_full[train_indices])

    # Select 200 images for test
    test_indices = np.random.choice(class_indices, 200, replace=False)

    # Append the test images and labels to the respective lists
    x_test.append(x_test_full[test_indices])
    y_test.append(y_test_full[test_indices])

    # Generate 100 augmented images for training
    x_augmented = data_gen.flow(x_train_full[train_indices], y_train_full[train_indices], batch_size=100)

    # Append the augmented images and labels to the respective lists
    x_train.append(x_augmented[0])
    y_train.append(x_augmented[1])

# Concatenate the list of images and labels to form the final training and test sets
x_train = np.concatenate(x_train)
y_train = np.concatenate(y_train)
x_test = np.concatenate(x_test)
y_test = np.concatenate(y_test)

print("training set shape: ", x_train.shape)
print("training label shape: ", y_train.shape)
print("test set shape: ", x_test.shape)
print("test label shape: ", y_test.shape)

但它一直说这样的错误：

IndexError: index 15753 is out of bounds for axis 0 with size 10000

【问题讨论】：

标签： tensorflow machine-learning conv-neural-network mnist data-augmentation

【解决方案1】：

您正在混合训练和测试集。在循环中，您从训练集中获取 class_indices：

# Get the indices of the images for this class
class_indices = np.where(y_train_full == class_n)[0]

但是你正在使用这些训练索引（可能是 10000 以上的数字！）来解决测试集中的索引（只有 10000 个样本）一些行进一步向下：

# Select 200 images for test
test_indices = np.random.choice(class_indices, 200, replace=False)

因此，您将需要对测试集循环中的标签进行相同的索引选择，它应该会成功。

【讨论】：