Image segmentation eval crashes with model_to_dataset on RELLIS-3D
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- machine-learning, testing
Research direction
Start with TorchImageSegmentationModel.eval() in perceptionmetrics/models/torch_segmentation.py and the confusion matrix and metrics table code in perceptionmetrics/utils/segmentation_metrics.py. Reproduce with examples/tutorial_image_segmentation_sample.ipynb using translation_direction="model_to_dataset" and a dataset with non-contiguous class indices. Add the proposed one-image regression test in tests/models/test_torch_segmentation.py; done means evaluation returns the metrics table without an IndexError and both translation directions give matching per-class IoU.
Written by the indexing model from the issue text.
Description
Problem
TorchImageSegmentationModel.eval() raises IndexError with translation_direction="model_to_dataset" when the dataset class indices are not contiguous, e.g. RELLIS-3D (20 classes, indices up to 34) or Cityscapes label IDs. It fails only after inference over the whole split. The GUI Evaluator hits the same error.
Steps to reproduce
- Run
examples/tutorial_image_segmentation_sample.ipynbup to "Init dataset and model objects". - Evaluate with:
results = model.eval(dataset, split="test", translation_direction="model_to_dataset")
Expected vs actual
Expected: the metrics table, matching dataset_to_model (class names map one to one).
Actual: IndexError once inference finishes.
Logs
RELLIS-3D run (the first line is a print I added):
dataset classes: 20, max idx: 34
[...]
Traceback (most recent call last):
[...]
return um.get_metrics_dataframe(metrics_factory, eval_ontology)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
results[class_name][metric] = float(per_class[class_data["idx"]])
~~~~~~~~~^^^^^^^^^^^^^^^^^^^
IndexError: index 23 is out of bounds for axis 0 with size 20
Screenshots
Root cause
The confusion matrix is sized by the class count, not the largest index:
With model_to_dataset, eval_ontology is the dataset ontology, so for RELLIS-3D the matrix is 20x20. GT pixels with index 20 or higher are dropped here:
and building the table then indexes past the end:
Proposed approach
- Size the matrix by the largest index, as
export()already does. Unused indices get NaN and the macro average skips them.
- Add
tests/models/test_torch_segmentation.py: a one-image dataset with indices 0, 3 and 4 and a model that returns the GT, expecting IoU 1.0 in both directions. It uses the torch skip guard fromtests/utils/test_torch.py. - Locally, both directions then give identical per-class IoU on the RELLIS-3D sample (mIoU 0.4617), and the suite passes.
- The same line exists in the LiDAR, TensorFlow and
eval_predspaths, which I could not run here. Should they go in the same PR?
Expected impact
model_to_dataset no longer crashes for non-contiguous dataset ontologies. Contiguous ontologies are unaffected (largest index plus one equals the class count).
Environment
- Commit: b1ad785d2cd505027458d467bf6bef7c33e6ce47 (master)
- OS: Ubuntu 26.04.1 LTS on WSL2, kernel 6.6.114.1-microsoft-standard-WSL2
- Python 3.12.14, torch 2.4.1+cpu, torchvision 0.19.1+cpu, numpy 1.26.4, streamlit 1.54.0
I'm happy to open a PR once the approach looks right to you.
cc @dpascualhe
- Dominant language
- Python
- Stars
- 114
- Forks
- 116
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 11
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from JdeRobot/PerceptionMetrics
-
pm_eval_model crashes with TypeError after the evaluation when --out_fname is not given (regression from #617)Possibly taken @Siddhanthguptaa claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
JdeRobot/PerceptionMetrics#675 ·
Maintainers usually reply within 1 day
-
Micro accuracy in segmentation is always the same as macro accuracyPossibly taken @guptakaran20 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
JdeRobot/PerceptionMetrics#674 · 1 comment ·
Maintainers usually reply within 1 day
-
Segmentation metrics give wrong results or crash for invalid predicted classesPossibly taken @soujasK claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
JdeRobot/PerceptionMetrics#659 · 4 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
[Bug] Inference tab renders normalized tensors without inverse normalization.Possibly taken @divyanshAgarwal123 claimed this 61 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
JdeRobot/PerceptionMetrics#524 · 8 comments ·
Maintainers usually reply within 1 day
-
Detection AUC-PR is underestimated: the curve is not anchored at recall 0, so a perfect detector scores (N-1)/NPossibly taken @Siddhanthguptaa claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 20/100
JdeRobot/PerceptionMetrics#672 ·
Maintainers usually reply within 1 day
All issues in JdeRobot/PerceptionMetrics
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
RunestoneInteractive/rs#1661 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
streamlink/streamlink#7192 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aws-samples/sample-per-model-bedrock#795 ·
Maintainers usually reply within 1 day