ValueError：数据基数不明确答案

【问题标题】：ValueError: Data cardinality is ambiguousValueError：数据基数不明确
【发布时间】：2020-09-26 21:36:22
【问题描述】：

我正在尝试用取自 DataFrame 的数据训练 LSTM 网络。

代码如下：

x_lstm=x.to_numpy().reshape(1,x.shape[0],x.shape[1])

model = keras.models.Sequential([
    keras.layers.LSTM(x.shape[1], return_sequences=True, input_shape=(x_lstm.shape[1],x_lstm.shape[2])),
    keras.layers.LSTM(NORMAL_LAYER_SIZE, return_sequences=True),
    keras.layers.LSTM(NORMAL_LAYER_SIZE),
    keras.layers.Dense(y.shape[1])
])

optimizer=keras.optimizers.Adadelta()

model.compile(loss="mse", optimizer=optimizer)
for i in range(150):
    history = model.fit(x_lstm, y)
    save_model(model,'tmp.rnn')

这失败了

ValueError: Data cardinality is ambiguous:
  x sizes: 1
  y sizes: 99
Please provide data which shares the same first dimension.

当我将模型更改为

model = keras.models.Sequential([
    keras.layers.LSTM(x.shape[1], return_sequences=True, input_shape=x_lstm.shape),
    keras.layers.LSTM(NORMAL_LAYER_SIZE, return_sequences=True),
    keras.layers.LSTM(NORMAL_LAYER_SIZE),
    keras.layers.Dense(y.shape[1])
])

它失败并出现以下错误：

Input 0 of layer lstm_9 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 1, 99, 1200]

如何让它工作？

x 的形状为 (99, 1200)（99 个项目，每个项目有 1200 个特征，这只是一个更大的数据集的样本），y 的形状为 (99, 1)

【问题讨论】：

尝试 x_lstm=x.to_numpy().reshape(x.shape[0],1,x.shape[1]) 与 lstm 中的 input_shape 等于 (x_lstm.shape[1],x_lstm .shape[2])
@MarcoCerliani 是的，可行

标签： python tensorflow keras lstm

【解决方案1】：

正如Error 所暗示的那样，X 和y 的First Dimension 是不同的。 First Dimension 表示Batch Size 应该是一样的。

请确保Y 也有shape、(1, something)。

我可以使用下面显示的代码重现您的错误：

from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM
import tensorflow as tf
import numpy as np


# define sequences
sequences = [
    [1, 2, 3, 4],
       [1, 2, 3],
             [1]
    ]

# pad sequence
padded = pad_sequences(sequences)
X = np.expand_dims(padded, axis = 0)
print(X.shape) # (1, 3, 4)

y = np.array([1,0,1])
#y = y.reshape(1,-1)
print(y.shape) # (3,)

model = Sequential()
model.add(LSTM(4, return_sequences=False, input_shape=(None, X.shape[2])))
model.add(Dense(1, activation='sigmoid'))

model.compile (
    loss='mean_squared_error',
    optimizer=tf.keras.optimizers.Adam(0.001))

model.fit(x = X, y = y)

如果我们观察Print 语句，

Shape of X is  (1, 3, 4)
Shape of y is (3,)

可以通过取消注释 y = y.reshape(1,-1) 行来修复此错误，这使得 X 和 y 的 First Dimension (Batch_Size) 等于 (1) .

现在，工作代码和输出如下所示：

from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM
import tensorflow as tf
import numpy as np


# define sequences
sequences = [
    [1, 2, 3, 4],
       [1, 2, 3],
             [1]
    ]

# pad sequence
padded = pad_sequences(sequences)
X = np.expand_dims(padded, axis = 0)
print('Shape of X is ', X.shape) # (1, 3, 4)

y = np.array([1,0,1])
y = y.reshape(1,-1)
print('Shape of y is', y.shape) # (1, 3)

model = Sequential()
model.add(LSTM(4, return_sequences=False, input_shape=(None, X.shape[2])))
model.add(Dense(1, activation='sigmoid'))

model.compile (
    loss='mean_squared_error',
    optimizer=tf.keras.optimizers.Adam(0.001))

model.fit(x = X, y = y)

以上代码的输出是：

Shape of X is  (1, 3, 4)
Shape of y is (1, 3)
1/1 [==============================] - 0s 1ms/step - loss: 0.2588
<tensorflow.python.keras.callbacks.History at 0x7f5b0d78f4a8>

希望这会有所帮助。快乐学习！

【讨论】：

【解决方案2】：

我也遇到了这个问题。请帮帮我。

ValueError: Data cardinality is ambiguous:
  x sizes: 770
  y sizes: 771
Make sure all arrays contain the same number of samples.

下面是我使用的代码。

import math

# split into train and test sets 
train_size = int((dataset.shape[0]*(6/8)))
test_size = len(dataset) - train_size

X_train = X[0:train_size] 
X_test = X[train_size:X.shape[0]]
y_train = y[0:train_size] 
y_test = y[train_size:X.shape[0]]

# %%
# Reshape input to 3D 
train_resize = int(X_train.shape[0]/n_steps)
test_resize = int(X_test.shape[0]/n_steps)

rows_train, cols_train = X_train.shape
rows_test, cols_test = X_test.shape
delrows_train = np.random.randint(rows_train, size=(7,1))
delrows_test = np.random.randint(rows_test, size=(16,1))
X_train = np.delete(X_train, delrows_train, axis=0)
X_test = np.delete(X_test, delrows_test, axis=0)

# print(rows_train)
print(train_resize)
print(test_resize)

X_train = X_train.reshape(train_resize,n_steps,X.shape[1])
y_train = y_train[::n_steps]
X_test = X_test.reshape(test_resize,n_steps,X.shape[1])
y_test = y_test[::n_steps]
#%% 

print("X_train size- ", X_train.shape)
print("y_train size- ", y_train.shape)
print("X_test size- ", X_test.shape)
print("y_test size- ", y_test.shape)

# design and train LSTM  network 
from keras.layers import Dropout
from tensorflow.keras.optimizers import Adam
import timeit
import keras

start = timeit.default_timer()

model = Sequential()
model.add(LSTM(100, input_shape=(X_train.shape[1], X_train.shape[2]),return_sequences=True))
model.add(LSTM(100,return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
opt = Adam(learning_rate=0.0001)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=100, batch_size=32, validation_data=(X_test, y_test), verbose=2, shuffle=False)

stop = timeit.default_timer()

【讨论】：