object - how to check both training/eval performances in tensorflow object_detection

Question

When I check the tensorboard for observing the training performance, there only shows the eval_0 (in blue) result.

While it should be a separate train (in orange) and eval (in blue) result as shown in the website of tensorboard (https://www.tensorflow.org/guide/summaries_and_tensorboard?).

However, I want to compare the model performance on training dataset and eval dataset.

So I checked the models/research/object_detection/model_main.py and want to know

if I I can get the precision based on the train and eval dataset by set the flag of model_dir to model/eval folder and set the flag of eval_training_data to model/train folder?

flags.DEFINE_string('model_dir', None, 'Path to output model directory '
                     'where event and checkpoint files will be written.')

flags.DEFINE_boolean('eval_training_data', False,
                     'If training data should be evaluated for this job. Note '
                     'that one call only use this in eval-only mode, and '
                     '`checkpoint_dir` must be supplied.')

And I'm confused with this sentence.

Note that one call only use this in eval-only mode, and checkpoint_dir must be supplied.

Does it means if I just want run it in eval-only mode, then I must set the checkpoint_dir? And if I want to run it with train and eval at the same time, I don't need to set the checkpoint_dir?

Answer 1

If you want to evaluate your model on validation data you should use:

python models/research/object_detection/model_main.py --pipeline_config_path=/path/to/pipeline_file --model_dir=/path/to/output_results --checkpoint_dir=/path/to/directory_holding_checkpoint --run_once=True

If you want to evaluate your model on training data, you should set 'eval_training_data' as True, that is:

python models/research/object_detection/model_main.py --pipeline_config_path=/path/to/pipeline_file --model_dir=/path/to/output_results --eval_training_data=True --checkpoint_dir=/path/to/directory_holding_checkpoint --run_once=True

---

I also add comments to clarify some of previous options:

--pipeline_config_path: path to "pipeline.config" file used to train detection model. This file should include paths to the TFRecords files (train and test files) that you want to evaluate, i.e. :

    ...
    train_input_reader: {
        tf_record_input_reader {
                #path to the training TFRecord
                input_path: "/path/to/train.record"
        }
        #path to the label map 
        label_map_path: "/path/to/label_map.pbtxt"
    }
    ...
    eval_input_reader: {
        tf_record_input_reader {
            #path to the testing TFRecord
            input_path: "/path/to/test.record"
        }
        #path to the label map 
        label_map_path: "/path/to/label_map.pbtxt"
    }
    ...

--model_dir: Output directory where resulting metrics will be written, particularly "events.*" files that can be read by tensorboard.

--checkpoint_dir: Directory holding a checkpoint. That is the model directory where checkpoint files ("model.ckpt.*") has been written, either during training process, or after export it by using "export_inference_graph.py".

--run_once: True to run just one round of evaluation.