tensorflow.python.framework.errors_impl.InternalError：GPU 同步失败答案

【问题标题】：tensorflow.python.framework.errors_impl.InternalError: GPU sync failedtensorflow.python.framework.errors_impl.InternalError：GPU 同步失败
【发布时间】：2022-04-14 01:37:15
【问题描述】：

我已经安装了以下：

窗口 10
Python 3.8
Tensorflow-GPU 2.3
库达 10.1
CudNN 7.6.5
英伟达 gtx 1080
驱动版本：451.48
内存：8192MiB

在训练过程中出现以下错误：

Traceback (most recent call last):
 File "training.py", line 519, in <module>
   history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, \
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
   return method(self, *args, **kwargs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1103, in fit
  callbacks.on_train_batch_end(end_step, logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 440, in on_train_batch_end
  self._call_batch_hook(ModeKeys.TRAIN, 'end', batch, logs=logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 289, in _call_batch_hook
  self._call_batch_end_hook(mode, batch, logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 309, in _call_batch_end_hook
  self._call_batch_hook_helper(hook_name, batch, logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 342, in _call_batch_hook_helper
  hook(batch, logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 961, in on_train_batch_end
   self._batch_update_progbar(batch, logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\callbacks.py", line 1016, in _batch_update_progbar
   logs = tf_utils.to_numpy_or_python_type(logs)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\utils\tf_utils.py", line 537, in to_numpy_or_python_type
  return nest.map_structure(_to_single_numpy_or_python_type, tensors)
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\util\nest.py", line 635, in map_structure
  structure[0], [func(*x) for x in entries],
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\util\nest.py", line 635, in <listcomp>
  structure[0], [func(*x) for x in entries],
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\keras\utils\tf_utils.py", line 533, in _to_single_numpy_or_python_type
   x = t.numpy()
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\framework\ops.py", line 1063, in numpy
  maybe_arr = self._numpy()  # pylint: disable=protected-access
 File "C:\Anaconda3_64\lib\site-packages\tensorflow\python\framework\ops.py", line 1031, in _numpy
  six.raise_from(core._status_to_exception(e.code, e.message), None)  # pylint: disable=protected-access
 File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: GPU sync failed

内部错误：GPU 同步失败

有什么线索吗？

【问题讨论】：

虽然你没提，但你好像在windows上。如果 GPU 内核运行时间超过 2 秒，您可以点击 CUDA_ERROR_LAUNCH_TIMEOUT。您不妨阅读this。您还会发现很多关于堆栈溢出的问题都在讨论这个问题。
@RobertCrovella 我已经更新了问题，我添加了更多细节
@RobertCrovella 我尝试将 WDDM TDR 延迟设置为 10 和 30，但仍然出现相同的错误。
stackoverflow.com/questions/51112126/…
调查various reports看来最常见的原因可能是你的GPU内存不足。

标签： python tensorflow nvidia

【解决方案1】：

请验证CUDA 和CUPTI 的路径设置正确，如下所示，以便在您的系统中启用GPU support。

SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include;%PATH%
SET PATH=C:\tools\cuda\bin;%PATH%

有时由于其他应用程序对GPU 的大量使用或处理大量输入数据，会出现GPU sync failed 错误。因此，您应该停止这些应用程序或笔记本并再次尝试执行您的代码。

【讨论】：