在 tensorflow 2 中训练我的模型检查点预计是基于对象的检查点

Question

我正在按照本教程 https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html 创建一个新模型，但是当我尝试训练模型时，我执行以下语句：

python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config

这是输出：

    2020-10-21 14:01:05.982155: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-21 14:01:06.000363: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbb44f076b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-21 14:01:06.000386: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
W1021 14:01:06.001132 4568120768 cross_device_ops.py:1202] There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
I1021 14:01:06.001378 4568120768 mirrored_strategy.py:341] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 10000
I1021 14:01:06.005095 4568120768 config_util.py:552] Maybe overwriting train_steps: 10000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I1021 14:01:06.005259 4568120768 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reading unweighted datasets: ['/Users/jose/Desktop/Tensorflow/workspace/training_demo/annotations/train.record']
I1021 14:01:06.044616 4568120768 dataset_builder.py:148] Reading unweighted datasets: ['/Users/jose/Desktop/Tensorflow/workspace/training_demo/annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['/Users/jose/Desktop/Tensorflow/workspace/training_demo/annotations/train.record']
I1021 14:01:06.045146 4568120768 dataset_builder.py:77] Reading record datasets for input file: ['/Users/jose/Desktop/Tensorflow/workspace/training_demo/annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I1021 14:01:06.045268 4568120768 dataset_builder.py:78] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W1021 14:01:06.045435 4568120768 dataset_builder.py:85] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:99: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W1021 14:01:06.047674 4568120768 deprecation.py:317] From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:99: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:221: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W1021 14:01:06.087840 4568120768 deprecation.py:317] From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:221: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W1021 14:01:12.571774 4568120768 deprecation.py:317] From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/inputs.py:262: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1021 14:01:15.363219 4568120768 deprecation.py:317] From /opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/inputs.py:262: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Traceback (most recent call last):
  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "/opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 104, in main
    model_lib_v2.train_loop(
  File "/opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/model_lib_v2.py", line 564, in train_loop
    load_fine_tune_checkpoint(detection_model,
  File "/opt/anaconda3/envs/ComputerVision/lib/python3.8/site-packages/object_detection/model_lib_v2.py", line 346, in load_fine_tune_checkpoint
    raise IOError('Checkpoint is expected to be an object-based checkpoint.')
OSError: Checkpoint is expected to be an object-based checkpoint.

我不知道为什么会引发以下错误 Checkpoint is expected to be an object-based checkpoint。如果我一步一步地按照教程进行操作。谁能帮帮我？

Answer 1

日志听起来像是 fine_tune_checkpoint 属性指向一个无效的检查点位置：

OSError: Checkpoint is expected to be an object-based checkpoint.

检查您的 pipeline.config 文件中的 fine_tune_checkpoint 属性，该文件位于：models/my_ssd_resnet50_v1_fpn/pipeline.config

在您链接到的示例中，它设置为：

fine_tune_checkpoint: "pre-trained-models/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/checkpoint/ckpt-0"

您是否将其设置为有效的检查点目录？

您需要一个目录，通常命名为 checkpoint，其中至少有 3 个有效的检查点文件：checkpoint、ckpt-0.index 和 ckpt-0.data-00000-of-00001。您使用它的 id 链接到给定的检查点：例如ckpt-0.

如果您需要下载预训练模型，另请参阅TensorFlow 2 Detection Model Zoo。

您链接到的示例使用了SSD ResNet50 V1 FPN 640x640。

Answer 2

尝试将fine_tune_checkpoint_type 更改为"detection"。

Answer 3

我认为您的配置文件有问题。请搜索配置文件fine_tune_checkpoint:

D:/faster_rcnn_resnet101_v1_800x1333_coco17_gpu-8/checkpoint/ckpt-0

还有这个值，类似于上面那行。

如果有机会分享你的配置数据，那会更有帮助。

Answer 4

我遇到了和你一样的问题，上面的建议对我也不起作用。

最后我意识到fine_tune_checkpoint 期待一条没有.index 扩展名的路径。但是，您从模型动物园复制的文件包含.index 扩展名，如果您直接从中复制了路径，那么这个.index 扩展名很可能会保留在那里，所以这会导致错误。

对我来说，将 fine_tune_checkpoint 从 ...checkpoint/ckpt-0.index 更改为 ...checkpoint/ckpt-0 已解决问题。

Answer 5

我认为您只需将fine_tune_checkpointckpt-0.index 更改为ckpt-0