为 Keras model.fit_generator 使用生成器

Question

我最初在编写用于训练 Keras 模型的自定义生成器时尝试使用 generator 语法。所以我从__next__yielded。但是，当我尝试使用 model.fit_generator 训练我的模式时，我会收到一个错误，即我的生成器不是迭代器。解决方法是将yield 更改为return，这也需要重新调整__next__ 的逻辑以跟踪状态。与让yield 为我做这些工作相比，这相当麻烦。

有没有办法让yield 工作？如果我必须使用return 语句，我将需要编写更多的迭代器，这些迭代器的逻辑必须非常笨拙。

Answer 1

由于您没有发布代码，我无法帮助您调试代码，但我缩写了我为语义分割项目编写的自定义数据生成器，供您用作模板：

def generate_data(directory, batch_size):
    """Replaces Keras' native ImageDataGenerator."""
    i = 0
    file_list = os.listdir(directory)
    while True:
        image_batch = []
        for b in range(batch_size):
            if i == len(file_list):
                i = 0
                random.shuffle(file_list)
            sample = file_list[i]
            i += 1
            image = cv2.resize(cv2.imread(sample[0]), INPUT_SHAPE)
            image_batch.append((image.astype(float) - 128) / 128)

        yield np.array(image_batch)

用法：

model.fit_generator(
    generate_data('~/my_data', batch_size),
    steps_per_epoch=len(os.listdir('~/my_data')) // batch_size)

Answer 2

我最近玩过 Keras 的生成器，我终于设法准备了一个示例。它使用随机数据，因此尝试教授 NN 毫无意义，但它很好地说明了使用 Keras 的 python 生成器。

生成一些数据

import numpy as np
import pandas as pd
data = np.random.rand(200,2)
expected = np.random.randint(2, size=200).reshape(-1,1)

dataFrame = pd.DataFrame(data, columns = ['a','b'])
expectedFrame = pd.DataFrame(expected, columns = ['expected'])

dataFrameTrain, dataFrameTest = dataFrame[:100],dataFrame[-100:]
expectedFrameTrain, expectedFrameTest = expectedFrame[:100],expectedFrame[-100:]

发电机

def generator(X_data, y_data, batch_size):

  samples_per_epoch = X_data.shape[0]
  number_of_batches = samples_per_epoch/batch_size
  counter=0

  while 1:

    X_batch = np.array(X_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
    y_batch = np.array(y_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
    counter += 1
    yield X_batch,y_batch

    #restart counter to yeild data in the next epoch as well
    if counter >= number_of_batches:
        counter = 0

Keras 模型

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten, Reshape
from keras.layers.convolutional import Convolution1D, Convolution2D, MaxPooling2D
from keras.utils import np_utils


model = Sequential()
model.add(Dense(12, activation='relu', input_dim=dataFrame.shape[1]))
model.add(Dense(1, activation='sigmoid'))


model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])

#Train the model using generator vs using the full batch
batch_size = 8

model.fit_generator(
    generator(dataFrameTrain,expectedFrameTrain,batch_size),
    epochs=3,
    steps_per_epoch = dataFrame.shape[0]/batch_size,
    validation_data = generator(dataFrameTest,expectedFrameTest,batch_size*2),
    validation_steps = dataFrame.shape[0]/batch_size*2
)

#without generator
#model.fit(
#    x = np.array(dataFrame),
#    y = np.array(expected),
#    batch_size = batch_size,
#    epochs = 3
#)

输出

Epoch 1/3
25/25 [==============================] - 3s - loss: 0.7297 - acc: 0.4750 - 
val_loss: 0.7183 - val_acc: 0.5000
Epoch 2/3
25/25 [==============================] - 0s - loss: 0.7213 - acc: 0.3750 - 
val_loss: 0.7117 - val_acc: 0.5000
Epoch 3/3
25/25 [==============================] - 0s - loss: 0.7132 - acc: 0.3750 - 
val_loss: 0.7065 - val_acc: 0.5000

Answer 3

这是我为读取任何大小的文件实现它的方式。它就像一个魅力。

import pandas as pd

hdr=[]
for i in range(num_labels+num_features):
    hdr.append("Col-"+str(i)) # data file do not have header so I need to
                              # provide one for pd.read_csv by chunks to work

def tgen(filename):
    csvfile = open(filename)
    reader = pd.read_csv(csvfile, chunksize=batch_size,names=hdr,header=None)
    while True:
    for chunk in reader:
        W=chunk.values        # labels and features
        Y =W[:,:num_labels]   # labels 
        X =W[:,num_labels:]   # features
        X= X / 255            # any required transformation
        yield X, Y
    csvfile = open(filename)
    reader = pd.read_csv(csvfile, chunksize=batchz,names=hdr,header=None)

我主要的后面

nval=number_of_validation_samples//batchz
ntrain=number_of_training_samples//batchz
ftgen=tgen("training.csv")
fvgen=tgen("validation.csv")

history = model.fit_generator(ftgen,
                steps_per_epoch=ntrain,
                validation_data=fvgen,
                validation_steps=nval,
                epochs=number_of_epochs,
                callbacks=[checkpointer, stopper],
                verbose=2)

Answer 4

我想用 TensorFlow 2.x 升级 Vaasha 的代码，以实现训练效率以及数据处理的简易性。这对于图像处理特别有用。

使用上例中 Vaasha 生成的 Generator 函数或使用 tf.data.dataset API 处理数据。后一种方法在处理任何带有元数据的数据集时非常有用。例如，MNIST 数据可以通过几条语句进行加载和处理。

import tensorflow as tf # Ensure that TensorFlow 2.x is used
tf.compat.v1.enable_eager_execution()
import tensorflow_datasets as tfds # Needed if you are using any of the tf datasets such as MNIST, CIFAR10
mnist_train = tfds.load(name="mnist", split="train")

使用 tfds.load 数据集。加载和处理数据后（例如，转换分类变量、调整大小等）。

现在使用 TensorFlow 2.x 升级 keras 模型

 model = tf.keras.Sequential() # Tensorflow 2.0 upgrade
 model.add(tf.keras.layers.Dense(12, activation='relu', input_dim=dataFrame.shape[1]))
 model.add(tf.keras.layers.Dense(1, activation='sigmoid'))


 model.compile(loss='binary_crossentropy',
               optimizer='adadelta',
               metrics=['accuracy'])

 #Train the model using generator vs using the full batch
 batch_size = 8

 model.fit_generator(generator(dataFrameTrain,expectedFrameTrain,batch_size),
                epochs=3,
                steps_per_epoch=dataFrame.shape[0]/batch_size,
                validation_data=generator(dataFrameTest,expectedFrameTest,batch_size*2),
                validation_steps=dataFrame.shape[0]/batch_size*2)

这将升级模型以在 TensorFlow 2.x 中运行