【发布时间】：2020-10-30 09:08:11
【问题描述】：
前段时间，我编写了一个检测猫和狗的 CNN 代码。数据取自 Kaggle，CNN 代码成功运行。在此之前，为了加快学习速度，我安装了 Nvidia Cuda。现在我从 Kaggle (https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia) 发现了一个新任务。我又写了CNN代码。但是我收到了一个我从未见过的错误。
代码：
import os
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Dense, Flatten
from keras import optimizers
from keras.preprocessing.image import ImageDataGenerator
def data():
dataset = 'C:/Users/Lenovo/Desktop/chest_xray/'
train = os.path.join(dataset, 'train')
test = os.path.join(dataset, 'test')
validation = os.path.join(dataset, 'val')
train_data = ImageDataGenerator(rescale=1. / 255)
test_data = ImageDataGenerator(rescale=1. / 255)
train_generator = train_data.flow_from_directory(
train,
target_size=(1000, 1000),
batch_size=16,
class_mode='binary')
validation_generator = test_data.flow_from_directory(
validation,
target_size=(1000, 1000),
batch_size=16,
class_mode='binary')
test_generator = test_data.flow_from_directory(
test,
target_size=(1000, 1000),
batch_size=16,
class_mode='binary')
for data_batch, labels_batch in train_generator:
print('data batch shape:', data_batch.shape)
print('label batch shape:', labels_batch.shape)
break
return validation_generator, test_generator, train_generator
def detector():
model = Sequential()
model.add(Conv2D(32, (4, 4), activation='relu', input_shape=(1000, 1000, 1)))
model.add(MaxPooling2D((3, 3)))
model.add(Conv2D(64, (4, 4), activation='relu'))
model.add(MaxPooling2D((3, 3)))
model.add(Conv2D(64, (4, 4), activation='relu'))
model.add(MaxPooling2D((3, 3)))
model.add(Conv2D(128, (4, 4), activation='relu'))
model.add(MaxPooling2D((3, 3)))
model.add(Conv2D(128, (4, 4), activation='relu'))
model.add(MaxPooling2D((3, 3)))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-4))
model.summary()
return model
validation_, test_, train_ = data()
model_ = detector()
history = model_.fit_generator(
train_,
steps_per_epoch=100,
epochs=30,
validation_data=validation_,
validation_steps=50)
输出：
C:\Users\Lenovo\anaconda3\python.exe "C:/Users/Lenovo/PycharmProjects/AI/1- Text.py"
2020-10-30 12:03:20.802426: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
Found 5216 images belonging to 2 classes.
Found 16 images belonging to 2 classes.
Found 624 images belonging to 2 classes.
data batch shape: (16, 1000, 1000, 3)
label batch shape: (16,)
2020-10-30 12:03:24.063892: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-10-30 12:03:25.009097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2020-10-30 12:03:25.009744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-10-30 12:03:25.017137: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:25.022144: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-10-30 12:03:25.024962: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-10-30 12:03:25.030998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-10-30 12:03:25.034229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-10-30 12:03:25.052663: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:25.053092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-30 12:03:25.053825: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-30 12:03:25.063634: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1dafce31430 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-30 12:03:25.064420: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-30 12:03:25.065036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2020-10-30 12:03:25.065861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-10-30 12:03:25.066244: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:25.066580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-10-30 12:03:25.066872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-10-30 12:03:25.067182: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-10-30 12:03:25.067619: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-10-30 12:03:25.068034: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:25.068448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-30 12:03:25.734018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-30 12:03:25.734464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-10-30 12:03:25.734664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-10-30 12:03:25.735009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2984 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-10-30 12:03:25.738356: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1daa2f2ca40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-30 12:03:25.738781: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1050 Ti, Compute Capability 6.1
WARNING:tensorflow:From C:/Users/Lenovo/PycharmProjects/AI/1- Text.py:66: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 997, 997, 32) 544
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 332, 332, 32) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 329, 329, 64) 32832
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 109, 109, 64) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 106, 106, 64) 65600
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 35, 35, 64) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 32, 32, 128) 131200
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 10, 10, 128) 0
_________________________________________________________________
conv2d_4 (Conv2D) (None, 7, 7, 128) 262272
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 2, 2, 128) 0
_________________________________________________________________
flatten (Flatten) (None, 512) 0
_________________________________________________________________
dense (Dense) (None, 256) 131328
_________________________________________________________________
dense_1 (Dense) (None, 256) 65792
_________________________________________________________________
dense_2 (Dense) (None, 1) 257
=================================================================
Total params: 689,825
Trainable params: 689,825
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
2020-10-30 12:03:27.997615: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-10-30 12:03:28.951268: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-10-30 12:03:29.803616: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.91GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-30 12:03:29.804253: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2020-10-30 12:03:29.805214: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:1115 : Not found: No algorithm worked!
Traceback (most recent call last):
File "C:/Users/Lenovo/PycharmProjects/AI/1- Text.py", line 66, in <module>
history = model_.fit_generator(
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1815, in fit_generator
return self.fit(
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1098, in fit
tmp_logs = train_function(iterator)
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
result = self._call(*args, **kwds)
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 840, in _call
return self._stateless_fn(*args, **kwds)
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2829, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
return self._call_flat(
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
outputs = execute.execute(
File "C:\Users\Lenovo\anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: No algorithm worked!
[[node sequential/conv2d/Conv2D (defined at /Users/Lenovo/PycharmProjects/AI/1- Text.py:66) ]] [Op:__inference_train_function_1397]
Function call stack:
train_function
2020-10-30 12:03:29.893708: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]
Process finished with exit code 1
我搜索了那个错误，但我没有解决这个问题。有什么解决办法吗？
【问题讨论】：
-
似乎是 Cuda 问题。错误源自here，又从here 调用。检查 Cuda 是否正确安装了正确的版本。此外，release 2.2.0 提到修复与环境变量
TF_DETERMINSTIC_OPS和TF_CUDNN_DETERMINISTIC相关的错误... -
正如我所说，我之前使用过 cuda 来检测猫和狗。我没有问题。你确定吗 ？这真的是关于 cuda 版本的吗？
-
不，我不确定它是关于 Cuda/Cudnn 版本的，我只是说你可以检查你的安装，或者只是重新安装它，并确保版本是正确的您的 TensorFlow 版本（例如，如果它应该是 9.1，请确保它不是 9.0 或 9.2），以防万一。似乎该问题与 Cuda/Cudnn 有关，因为这就是错误的来源。如果您没有最新版本，您也可以考虑更新 GPU 驱动程序，我过去遇到过一些问题，后来得到了解决。
-
这个问题可能是因为 GPU 内存耗尽了内存。尝试设置 memory_growth=True。在您的代码中附加以下 sn-p 并尝试。
import tensorflow as tf physical_devices = tf.config.list_physical_devices('GPU') tf.config.experimental.set_memory_growth(physical_devices[0], True).
标签： python-3.x tensorflow keras