使用张量板回调时出现 CUPTI 错误

Question

我目前正在使用 CPU 来训练 LSTM 神经网络。当我运行代码时

model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy']
)

tensorboard = TensorBoard(log_dir="logs/{}".format(NAME))

filepath = "RNN_Final-{epoch:02d}-{val_acc:.3f}"  # unique file name that will include the epoch and the validation acc for that epoch
checkpoint = ModelCheckpoint("models/{}.model".format(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')) # saves only the best ones

# Train model
history = model.fit(
    train_x, train_y,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    validation_data=(validation_x, validation_y),
    callbacks=[tensorboard, checkpoint],
)

# Score model
score = model.evaluate(validation_x, validation_y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
# Save model
model.save("models/{}".format(NAME))

它会引发以下 CUPTI 错误

2019-12-28 20:31:45.798191: E tensorflow/core/platform/default/device_tracer.cc:70] CUPTI error: CUPTI could not be loaded or symbol could not be found.
Traceback (most recent call last):
  File "e:/Tutorial_codes/Deep Learning/crypto_rnn_tut.py", line 163, in <module>
    callbacks=[tensorboard, checkpoint]
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 727, in fit
    use_multiprocessing=use_multiprocessing)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_arrays.py", line 675, in fit
    steps_name='steps_per_epoch')
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_arrays.py", line 405, in model_iteration
    callbacks._call_batch_hook(mode, 'end', batch_index, batch_logs)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\callbacks.py", line 236, in _call_batch_hook
    batch_hook(batch, logs)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\callbacks.py", line 519, in on_train_batch_end
    self.on_batch_end(batch, logs=logs)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\callbacks_v1.py", line 362, in on_batch_end
    profiler.save(self.log_dir, profiler.stop())
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\eager\profiler.py", line 140, in save
    gfile.MakeDirs(plugin_dir)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 438, in recursive_create_dir
    recursive_create_dir_v2(dirname)
  File "C:\Users\Aryan Soni\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 453, in recursive_create_dir_v2
    pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(path))
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: logs/60-SEQ-3-PRED-1577545294\plugins\profile\2019-12-28_20-31-45; No such file or directory

但是当我从model.fit 中删除callbacks 行时，一切正常，模型开始训练。

我是 Tensorflow 的新手，所以请告诉我为什么我不能使用 tensorboard，你需要 gpu 才能使用 tensorboard？

Answer 1

将此行放在您的张量板初始化行之前：

os.makedirs(f"logs\{NAME}", exist_ok=True)

您尚未创建 tensorboard 将记录到的文件夹。