TensorFlow 错误：ValueError("形状 %s 和 %s 不兼容"% (self, other))答案

【问题标题】：TensorFlow Error: ValueError("Shapes %s and %s are incompatible" % (self, other))TensorFlow 错误：ValueError("形状 %s 和 %s 不兼容"% (self, other))
【发布时间】：2020-08-11 08:42:17
【问题描述】：

我正在尝试使用 categorical cross-entropy 作为损失函数将 PCB 图像分为两类（defected 和 undefected）。相同的代码如下：

import numpy as np
import matplotlib.pyplot as plt

import tensorflow
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint


from keras.applications.resnet50 import preprocess_input
from keras.preprocessing.image import ImageDataGenerator

from sklearn.model_selection import train_test_split

def create_compiled_model():
  model = Sequential()
  model.add(ResNet50(include_top=False, weights=RESNET50_WEIGHTS, input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3), pooling=RESNET50_POOLING_AVERAGE))
  model.add(Dense(NUM_CLASSES, activation=DENSE_LAYER_ACTIVATION))
  model.layers[0].trainable = False

  sgd = SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
  model.compile(optimizer = sgd, loss = OBJECTIVE_FUNCTION, metrics = LOSS_METRICS)

  return model

def data_splitor():
  x = np.load("/content/data/xtrain.npy")
  y = np.load("/content/data/ytrain.npy")

  # Getting the Test and Train splits
  x_train, x_test, y_train, y_test = train_test_split(x, y, test_size= TRAIN_TEST_SPLIT, shuffle= True)

  # Getting the Train and Validation splits
  x__train, x__valid, y__train, y__valid = train_test_split(x_train, y_train, test_size= TRAIN_TEST_SPLIT, shuffle= True)

  return x__train, x__valid, x_test, y__train, y__valid, y_test

def data_generator(x, y, batch_size, seed=None, shuffle=True):
  data_generator = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=180, brightness_range=[0.3, 1.0], preprocessing_function=preprocess_input)
  generator = data_generator.flow(x_train, y_train, batch_size= batch_size, seed= seed, shuffle=shuffle)
  return generator

def run_program():
  x_train, x_valid, x_test, y_train, y_valid, y_test = data_splitor()
  train_generator = data_generator(x_train, y_train, BATCH_SIZE_TRAINING)
  validation_generator = data_generator(x_valid, y_valid, BATCH_SIZE_VALIDATION)

  cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
  cb_checkpointer = ModelCheckpoint(filepath = '/content/model/best.hdf5', monitor = 'val_loss', save_best_only = True, mode = 'auto')

  model = create_compiled_model()

  fit_history = model.fit_generator(
        train_generator,
        steps_per_epoch=STEPS_PER_EPOCH_TRAINING,
        epochs = NUM_EPOCHS,
        validation_data=validation_generator,
        validation_steps=STEPS_PER_EPOCH_VALIDATION,
        callbacks=[cb_checkpointer, cb_early_stopper]
  )

  plt.figure(1, figsize = (15,8)) 

  plt.subplot(221)
  plt.plot(fit_history.history['acc'])  
  plt.plot(fit_history.history['val_acc'])  
  plt.title('model accuracy')  
  plt.ylabel('accuracy')  
  plt.xlabel('epoch')  
  plt.legend(['train', 'valid']) 

  plt.subplot(222)  
  plt.plot(fit_history.history['loss'])  
  plt.plot(fit_history.history['val_loss'])  
  plt.title('model loss')  
  plt.ylabel('loss')  
  plt.xlabel('epoch')  
  plt.legend(['train', 'valid']) 

  plt.show()


  # Testing
  test_generator = data_generator(x_test, y_test, BATCH_SIZE_TESTING, 123, False)
  test_generator.reset()

  model.load_weights("/content/model/best.hdf5")
  pred = model.predict_generator(test_generator, steps = len(test_generator), verbose = 1)

  predicted_class_indices = np.argmax(pred, axis = 1)


# Running the program
try:
  with tensorflow.device('/device:GPU:0'):
    run_program()
except RuntimeError as e:
  print(e)

执行此操作后，我得到如下所示的 ValueError：

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:951 run  **
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:533 train_step  **
        y, y_pred, sample_weight, regularization_losses=self.losses)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:204 __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:143 __call__
        losses = self.call(y_true, y_pred)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:246 call
        return self.fn(y_true, y_pred, **self._fn_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:1527 categorical_crossentropy
        return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:4561 categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1117 assert_is_compatible_with
        raise ValueError("Shapes %s and %s are incompatible" % (self, other))

    ValueError: Shapes (None, 1) and (None, 2) are incompatible

我已经查看了this、this 和this，但无法解决错误。

非常感谢在解决此问题方面的帮助。

感谢普拉文

这是完整的回溯...link

【问题讨论】：

标签： python tensorflow machine-learning keras deep-learning

【解决方案1】：

似乎您的 y_train 数据具有形状 (None,1) 而您的网络正在预期 (None,2)。有两种方法可以解决这个问题：

1) 将模型输出更改为 1 个单位并将损失更改为二元交叉熵

或

2) 将 y_train 数据更改为分类数据。见this

如果您可以在此处发布您的 model.summary() 和您的数据集形状，它将帮助我们为您提供帮助。

【讨论】：

非常感谢。将 y_train 更改为分类数据解决了我的错误。

【解决方案2】：

您的回溯链接无效。但是，请尝试将分类交叉熵替换为二元交叉熵，因为您只有两个类。

【讨论】：

感谢您查看问题 :) 我尝试过binary cross entropy，但没有任何区别。
@PraveenRaj 您为此找到了任何修复方法吗？请分享

【解决方案3】：

我有同样的问题，但相反，我使用标签从TFRecord文件中解码到Int64格式，从'patporicalcrossentropy'改变我的损失函数，以“sparsecategoricalcrossentropy”解决问题。

【讨论】：

你能解释一下为什么这有帮助吗？我为我的花瓣笔记本做了同样的事情......而且它有效！
官方tensorflow文档：Use this crossentropy metric when there are two or more label classes. We expect labels to be provided as integers. If you want to provide labels using one-hot representation, please use CategoricalCrossentropy metric. There should be # classes floating point values per feature for y_pred and a single floating point value per feature for y_true. span>

【解决方案4】：

我遇到了类似的问题，上述这些解决方案都不起作用。我们得到这个错误的主要原因是我们未能在 X_train 和 Y_train 之间建立数据的 1 对 1 映射。这意味着 Y_train 的形状应该类似于 (No_of_Sequnces, no_of_classes)。

例子-

假设我的数据集有 2000 行和 5 个特征。其中 1 个序列 = 100 行数据。所以在重塑之前 x_train 会是这样的

X_train.shape = (2000,5)

在输入 LSTM 之前，我们应该（通常）将其重塑为 3D，因此

X_train.reshape(20,100,5)

另一方面，我们的 Y_Train 将是最初的。（如果是2D的，通过flattening变成1D）

Y_train.shape = (2000, )

所以，在输入 LSTM 之前，我们应该改变 Y_train 的形状

Y_train.shape =(20, 5)

20 将与训练集进行 1:1 映射，而 5 将与分类模型的最终密集层进行映射，我们应该使用分类交叉熵。

另外请注意，Y_train 应该是 2D 形状。那么我们如何重新塑造它呢？

检查 _train 数据的情况
如果在字符串中使用 one-hot 表示
如果每个类都是整数，则转换为分类 (refer)
更改为分类后再次参考 Y_train。

如果类数和列数相等，则使用以下代码将行数减少到 20（类似于 X_train）

for eachRowTemp in range(df_Y_Labels.__len__()):
   if(eachRowTemp%20 == 1):    
      Y_Label_Array.append(df_Y_Labels.loc[eachRowTemp])
 Y_Label = np.asarray(Y_Label_Array)

这应该可行。你也应该以类似的方式更改 Y_test。

【讨论】：