在我自己的数据集上运行 TensorFlow 对象检测训练作业的问题

Question

我可能遗漏了一些明显的东西，但是按照running locally README 中概述的步骤后，我无法在 EC2 V100 实例中成功提交训练作业。

到目前为止，我完成了以下步骤：

Tensorflow 版本'1.13.1'

1. 将训练和测试转换为 TFRecord 格式

2. 为我的数据集创建了一个包含 6 个类的新标签映射 pb.txt。

3. 更新了管道配置 file 以反映路径和类的数量。

我的最终目录结构如下（+表示文件夹，-表示文件）：

+ models
 + faster_rcnn_resnet101_coco_2018_01_28
   - model.ckpt.data-00000-of-00001
   - model.ckpt.meta
   - model.ckpt.index
 + model
    + train
    + eval
    - pipeline.config

+ data
 - train.record
 - test.record
 - tp_label_map.pbtxt

一个问题是我不知道模型中的 train 和 eval 文件夹对应于自述文件中的内容。

4. 填充环境变量并启动训练作业，如here 所示。

PIPELINE_CONFIG_PATH=/home/ubuntu/models/research/object_detection/models/faster_rcnn_resnet101_coco_2018_01_28/pipeline.config
MODEL_DIR=/home/ubuntu/models/research/object_detection/models/model
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --num_train_steps=${NUM_TRAIN_STEPS} \
    --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
    --alsologtostderr

我收到以下警告，它只在那里挂了 10 分钟左右。没有移动到火车阶段。

但我确实在模型目录中填充了文件（train 和 eval 为空）。

+models
  - events.out.tfevents.1557175306.ip-172-31-32-179
  - graph.pbtxt 
  - model.ckpt-0.data-00000-of-00001  
  - model.ckpt-0.index 
  - model.ckpt-0.meta

如果你查看评论here，但当我查看nvidia-smi 或tensorboard 时，我没有看到任何生成。

张量板输出

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

*********** In model lib ************* /home/ubuntu/models/research/object_detection/models/faster_rcnn_resnet101_coco_2018_01_28/pipeline.config
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f6c5b26d048>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/builders/dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/utils/ops.py:472: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/inputs.py:320: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/builders/dataset_builder.py:152: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1624: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py:2298: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/core/losses.py:345: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/eval_util.py:785: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/models/research/object_detection/utils/visualization_utils.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
    tf.py_function, which takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.

WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.

Answer 1

文档确实对model_dir有点不清楚，但是源码comment有明确的解释。

所以model_dir是你保存新检查点文件的目录，它与你用于微调的预训练检查点文件不同，你不应该设置@ 987654325@ 到预训练的检查点路径。

每次提交新的训练作业时最好保持model_dir为空，否则如果有检查点文件，模型可能会跳过训练（here）。

这里列出了train 和eval 目录以供说明。可以选择这样设置目录结构，但不必相同。您只需将一个空目录传递给model_dir 即可保存检查点文件。