【发布时间】：2019-10-02 18:49:21
【问题描述】：
我正在使用 Mobilenet-SSD-v2 训练一个模型，它训练了一段时间，然后尝试评估并冻结。
我在 tensorflow/tensorflow:latest-gpu docker 镜像中运行 tensorflow-gpu 1.14。我在 ubuntu 19.04 上使用 RTX 2060。我正在使用来自这个 git repo 的最新对象检测 API：https://github.com/tensorflow/models。
我尝试在model_lib.py 中设置throttle_secs，它没有做任何事情。我仍然可以训练，但每次尝试评估时，我都需要重新启动 docker 容器。
我只使用 git repo 提供的代码。我使用以下命令开始训练。
PIPELINE_CONFIG_PATH=/tensorflow/models/research/face/pipeline.config
MODEL_DIR=/tensorflow/models/research/face/training/
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python object_detection/model_main.py \
--pipeline_config_path=${PIPELINE_CONFIG_PATH} \
--model_dir=${MODEL_DIR} \
--num_train_steps=${NUM_TRAIN_STEPS} \
--sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
--alsologtostderr
我预计它会继续训练。但相反，我只是卡住了，需要重新启动。
I1002 18:28:30.106040 139663203059520 evaluation.py:255] Starting evaluation at 2019-10-02T18:28:30Z
I1002 18:28:30.717183 139663203059520 monitored_session.py:240] Graph was finalized.
2019-10-02 18:28:30.717937: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:0a:00.0
2019-10-02 18:28:30.718232: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-10-02 18:28:30.718251: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-10-02 18:28:30.718263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-10-02 18:28:30.718279: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-10-02 18:28:30.718295: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-10-02 18:28:30.718309: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-10-02 18:28:30.718326: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-10-02 18:28:30.718401: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.718861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-10-02 18:28:30.718888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-02 18:28:30.718898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-10-02 18:28:30.718907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-10-02 18:28:30.718992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.719242: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-02 18:28:30.719460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4946 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:0a:00.0, compute capability: 7.5)
I1002 18:28:30.720419 139663203059520 saver.py:1280] Restoring parameters from /tensorflow/models/research/face/training/model.ckpt-10756
I1002 18:28:32.285661 139663203059520 session_manager.py:500] Running local_init_op.
I1002 18:28:32.408489 139663203059520 session_manager.py:502] Done running local_init_op.
【问题讨论】：
-
几次迭代后它会停止吗？像在 100 到 500 之间？
-
大约 600 次迭代，然后冻结。
标签： python tensorflow object-detection-api