Benchmark results not consistent for Mask R-CNN Spinenet-49
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 15/100
- issue の種類
- バグ
- 明瞭さ
- 説明が足りない
- 活発さ
- 停滞
- 技術スタック
- python
調査の方向性
Start from official/detection/main.py in --mode=eval and the config configs/spinenet/spinenet49_mrcnn.yaml, then check how the evaluation batches and COCO annotation JSON are loaded. The report gives no concrete code change, only reproduced AP values (39.2 box, 35.0 mask) below the model zoo numbers (42.8, 37.8) for SpineNet-49 and SpineNet-143. Done would mean either reproducing the reported numbers or finding the cause, which needs the checkpoints and a GPU setup.
索引モデルが issue の本文から書いたものです。
説明
Hi,
I am trying to reproduce the AP scores reported for the Mask R-CNN model with Spinenet-49 and Spinenet-96 backbones. When I run main.py on eval mode, the results I got are lower than expected (results reported on MODEL ZOO). Here are the results I got for Spinenet-49 backbone:
Evaluate annotation type *bbox*
DONE (t=54.10s).
`Accumulating evaluation results...
DONE (t=8.79s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.392
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.571
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.427
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.162
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.438
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.608
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.337
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.529
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.565
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.370
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.603
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.71
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=59.38s).
Accumulating evaluation results...
DONE (t=8.46s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.350
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.549
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.375
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.131
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.396
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.56
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.311
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.477
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.506
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.305
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.552
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.674
In MODEL ZOO, the box AP and mask AP were 42.8 and 37.8. The results I got are 39.2 and 35.0.
To run the evaluation, here is the command I used:
MODEL_DIR=<path to directory with checkpoint files>
DATA_DIR=<path to directory with tfrecord files for COCO 2017 validation set>
EVAL_DIR=<path to instances_val2017.json>
MODE="eval"
CONFIG_PATH=<path to tpu directory>/tpu/models/official/detection/configs/spinenet/spinenet49_mrcnn.yaml
CHECKPOINT_PATH=<path to directory with checkpoint files>/model.ckpt
BATCH_SIZE=8
export PYTHONPATH=$PYTHONPATH:<path to tpu directory>/tpu/models/official/efficientnet
export CUDA_VISIBLE_DEVICES=0
python3 official/detection/main.py \
--model="mask_rcnn" \
--model_dir=$MODEL_DIR \
--checkpoint_path=$CHECKPOINT_PATH \
--mode=$MODE \
--use_tpu=False \
--export_to_tpu=False \
--config_file=$CONFIG_PATH \
--params_override="{train: {train_batch_size: $BATCH_SIZE}, eval: {val_json_file: $EVAL_DIR, eval_file_pattern: $DATA_DIR, eval_batch_size: $BATCH_SIZE, eval_samples: 5000}}"
I run the evaluation code on my local machine. My specs are:
- GPU: NVIDIA RTX 2080-TI
- Tensorflow version: 2.4.1
- CUDA version: 11.0
- Branch: r2.4
I used the checkpoints given in MODEL ZOO.
There is also a problem with the Spinenet-143 backbone. The results I got are lower than the Spinenet-49 backbone and inconsistent with the scores reported in MODEL ZOO. The supplied tar file misses the file "checkpoint" also. The results for Spinenet-143 bakckbone is as follows:
Evaluate annotation type *bbox*
DONE (t=57.48s).
Accumulating evaluation results...
DONE (t=8.53s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.384
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.547
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.415
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.168
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.469
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.646
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.351
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.551
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.599
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.423
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.627
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.743
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=61.33s).
Accumulating evaluation results...
DONE (t=8.39s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.345
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.531
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.374
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.146
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.422
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.595
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.323
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.500
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.541
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.364
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.570
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.691
Is there a problem with the checkpoints provided? Any help would be appreciated. Thanks!
- 主要言語
- Jupyter Notebook
- スター
- 5.3k
- フォーク
- 1.7k
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
tensorflow/tpu のほかの issue
-
難易度 2/5 半日 初心者へのやさしさ 62/100
tensorflow/tpu#3542 ·
-
imagenet_to_gcs.pyオープン
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
tensorflow/tpu#951 · リアクション 2 件 ·
-
Broken linkオープン
難易度 1/5 1時間未満 初心者へのやさしさ 70/100
tensorflow/tpu#908 ·
-
難易度 1/5 1時間未満 初心者へのやさしさ 62/100
tensorflow/tpu#889 · コメント 1 件 · リアクション 1 件 ·
-
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
tensorflow/tpu#582 · コメント 1 件 ·
似ている issue
-
needs-triage
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
flashinfer-ai/flashinfer#9803 ·
メンテナーはふだん 1 日以内に返信
-
bug module: rl
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
NVIDIA/Megatron-LM#9644 ·
メンテナーはふだん 2 日以内に返信
-
`build_flavor_config_from_local_checkpoint` fails with transformers 5.23.0: pipeline `impl` is now a dotted-path string対応中かも @mlflow-app が今日担当しました。 オープンhas-closing-pr nightly triage: reproduced
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
mlflow/mlflow#28000 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
X-PLUG/MobileAgent#304 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
FluxML/Optimisers.jl#254 ·