无效参数：在 TensorFlow 中调用 model.fit 时，indices[74,7] = 3298 不在第二个 Epoch 的 [0, 1827) 中

Question

我正在尝试使用 TensorFlow 2 创建 NLP

这是我的代码

def unicode_to_ascii(s):
    return ''.join(c for c in unicodedata.normalize('NFD', s)
                   if unicodedata.category(c) != 'Mn')


def preprocess_eng(w):
    w = unicode_to_ascii(w.lower().strip())

    w = re.sub(r"([?.!,])", r" \1 ", w)
    w = re.sub(r'[" "]+', " ", w)

    w = re.sub(r"[^a-zA-Z?.!,]+", " ", w)
    w = w.rstrip().strip()

    return w

def preprocess_chinese(w):
    w = unicode_to_ascii(w.lower().strip())
    w = re.sub(r'[" "]+', "", w)
    w = w.rstrip().strip()
    w = " ".join(list(w))  # add the space between words
    w = '<start> ' + w + ' <end>'
    return w

en_words, zh_words = create_dataset(path_to_data, max_examples=max_examples)
en_train, en_test, zh_train, zh_test = train_test_split(en_words, zh_words, test_size=0.1)

def max_length(data):
  max_length = max([len(x.split(' ')) for x in data])
  return max_length

# Training Data
max_length_eng_train = max_length(en_train)
max_length_zh_train = max_length(zh_train)

# Test Data
max_length_eng_test = max_length(en_test)
max_length_zh_test = max_length(zh_test)

en_tokenizer = Tokenizer()
en_tokenizer.fit_on_texts(en_train)

en_word_to_index = en_tokenizer.word_index
vocab_size_source = len(en_word_to_index) + 1

en_train = en_tokenizer.texts_to_sequences(en_train)
en_train = pad_sequences(en_train, maxlen=max_length_eng_train, padding='post')

en_test = en_tokenizer.texts_to_sequences(en_test)
en_test = pad_sequences(en_test, maxlen=max_length_eng_train, padding='post')


zh_tokenizer = Tokenizer()
zh_tokenizer.fit_on_texts(zh_train)

zh_word_to_index = zh_tokenizer.word_index
vocab_size_target = len(zh_word_to_index) + 1

zh_train = zh_tokenizer.texts_to_sequences(zh_train)
zh_train = pad_sequences(zh_train, maxlen=max_length_zh_train, padding='post')

zh_test = en_tokenizer.texts_to_sequences(zh_test)
zh_test = pad_sequences(zh_test, maxlen=max_length_zh_train, padding='post')

en_train = np.array(en_train)
zh_train = np.array(zh_train)
en_test = np.array(en_test)
zh_test = np.array(zh_test)

这就是为什么我从

vocab_size_source, vocab_size_target
(4195, 1827)

然后我尝试创建图层和模型

from attention import AttentionLayer
from keras import backend as K

K.clear_session() 
latent_dim = 256

# Encoder 
encoder_inputs = Input(shape=(max_length_eng_train,)) 
enc_emb = Embedding(vocab_size_source, latent_dim,trainable=True)(encoder_inputs)

#LSTM 1 
encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True) 
encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)

#LSTM 2 
encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True) 
encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)

#LSTM 3 
encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True) 
encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)

# Set up the decoder. 
decoder_inputs = Input(shape=(None,)) 
dec_emb_layer = Embedding(vocab_size_target, latent_dim,trainable=True) 
dec_emb = dec_emb_layer(decoder_inputs)

#LSTM using encoder_states as initial state
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True) 
decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])

#Attention Layer
attn_layer = AttentionLayer(name='attention_layer') 
attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])

# Concat attention output and decoder LSTM output 
decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])

#Dense layer
decoder_dense = TimeDistributed(Dense(vocab_size_target, activation='softmax')) 
decoder_outputs = decoder_dense(decoder_concat_input)

# Define the model
model = Model([encoder_inputs, decoder_inputs], decoder_outputs) 

model.compile(optimizer='rmsprop',
              loss='sparse_categorical_crossentropy', 
              metrics=['accuracy'])

然后我用这些函数训练我的模型

early_stopping = EarlyStopping(monitor='val_loss', mode='min', verbose=1)

EPOCH_NUMBER = 50
BATCH_SIZE = 128

history = model.fit([en_train, zh_train[:,:-1]], 
zh_train.reshape(zh_train.shape[0], zh_train.shape[1],1)[:,1:], 
                epochs=EPOCH_NUMBER, 
                callbacks=[early_stopping],
                batch_size=BATCH_SIZE,
                validation_data = ([en_test, zh_test[:,:-1]],           
                                    zh_test.reshape(zh_test.shape[0], zh_test.shape[1], 1)[:,1:]))

但是在完成第一个 Epoch 后我得到了这个错误

Invalid argument:  indices[74,7] = 3298 is not in [0, 1827)

我该如何解决这个问题，我错了哪一部分？

谢谢！

Answer 1

在您的情况下，最大索引是3298（用于目标），这当然意味着有多个1827 字。

您是否有可能在标记化和预处理之前获得3298 的长度，从而在您删除词汇表中的一些单词之前？

本质上会发生这样的事情：

vocab_size_target = len(zh_word_to_index) + 1

可能还有目标，不反映数据集的真实长度。

vocab_size_source, vocab_size_target
(4195, 1827), here 1827 < 3298

我会检查create_dataset(path_to_data, max_examples=max_examples) 方法来解决这个问题。