向量化文本作为 RNN 的输入

Question

我有以下函数可以向我的数据框添加一个新列。 我想将矢量化文本用作我的 RNN，但是，我无法重塑列以将其用作输入。我该如何解决这个问题？谢谢

# vectorization
max_length = 500
def vectorization(text):
  seq = text.split()
  if seq:
    vectorizer = TfidfVectorizer()
    vectorizer.fit(seq)
    vector = vectorizer.transform(seq)
    return sequence.pad_sequences(vector.toarray(), maxlen=max_length)
  else:
    print(seq)
    return seq

df['text_vector']=df['text_cleaned'].apply(vectorization)

X_train, X_test, Y_train, Y_test = train_test_split(df['text_vector'], df['sentiment'], train_size=0.80, shuffle=True)

X_train = X_train.to_numpy()
X_test = X_test.to_numpy()
Y_train = Y_train.to_numpy()
Y_test = Y_test.to_numpy()

X_train = X_train.reshape((X_train.shape[0], 500, 1))

此处出错：

ValueError: cannot reshape array of size 3876 into shape (3876,500,1)

Answer 1

几点

• 理想情况下，您应该将 TfidfVectorizer 放在完整的火车文本中，但不要像您正在做的那样按行显示
• pad_sequences 之后的每一行都是一个大小为 500 的 np 数组。因此，您必须明智地连接所有 np 数组行以创建大小为 (n X 500) 的 np 数组，其中 n 是 len(df)

固定代码（内嵌注释）：

from sklearn.feature_extraction.text import TfidfVectorizer
from keras.preprocessing import sequence


max_length = 500
def vectorization(vectorizer, text):
    vector = vectorizer.transform(text)
    return sequence.pad_sequences(vector.toarray(), maxlen=max_length)

import pandas as pd 
df = pd.DataFrame( {'text_cleaned': [
                                     'a cat on a table', 
                                     'a dog under a table', 
                                     'apple is red', 
                                     'sky is blue'] })
v = TfidfVectorizer()
# Fit on full test data text
v.fit(df['text_cleaned'])

df['text_vector']= df['text_cleaned'].apply(lambda text: vectorization(v, [text]))
# concatenate all the 500 length sequences
x_train = np.concatenate(df['text_vector'])
# reshape or use expand_dim to add last dimention so that it can be passed to RNN
x_train = x_train.reshape(-1,500,1)