COCO crowd annotations (iscrowd=1) are counted as ground truth, lowering detection recall and AP

Open Beginner friendly
  • Is something wrong?
#681 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
computer-vision

Research direction

Read perceptionmetrics/datasets/coco.py, especially CocoDataset.read_annotation(), to see how COCO annotations are converted. Add or run a test with both a normal annotation and one marked iscrowd=1; verify the crowd annotation does not count as a missed object and that the normal detection still scores correctly. The issue also mentions optionally ignoring predictions that match crowd regions, so check the project’s preferred evaluation behavior.

Written by the indexing model from the issue text.

Description

Description

CocoDataset.read_annotation() (perceptionmetrics/datasets/coco.py) returns every COCO annotation, including the ones with iscrowd=1. Nothing else in the repo checks iscrowd, so crowd regions (for example a group of people annotated as one region) are treated as normal objects.

The official COCO evaluation (pycocotools) ignores crowd regions: a missed crowd region is not a false negative, and a detection on one is not a false positive. Here they become false negatives, so recall and AP go down for classes that have crowd annotations, such as person.

Steps to reproduce

One image with a normal person box and a crowd region, and a perfect prediction on the person:

import numpy as np
from perceptionmetrics.utils.detection_metrics import DetectionMetricsFactory

anns = [
    {"bbox": [10, 10, 50, 100], "category_id": 1, "iscrowd": 0},
    {"bbox": [200, 20, 300, 150], "category_id": 1, "iscrowd": 1},  # crowd region
]
gt_boxes = np.array([[x, y, x + w, y + h] for x, y, w, h in (a["bbox"] for a in anns)], float)
gt_labels = np.array([a["category_id"] for a in anns])
pred_boxes = np.array([[10, 10, 60, 110]], float)
pred_labels, pred_scores = np.array([1]), np.array([0.9])

mf = DetectionMetricsFactory(iou_threshold=0.5)
mf.update(gt_boxes, gt_labels, pred_boxes, pred_labels, pred_scores)
m = mf.compute_metrics()[1]
print(m["TP"], m["FN"], m["Recall"], m["AP"])
Output

On master (b190ec4):

TP=1 FN=1 Recall=0.50 AP=0.545   # crowd region included, as read_annotation returns it today
TP=1 FN=0 Recall=1.00 AP=1.000   # same image with the crowd annotation left out

The crowd region counts as a missed object, so recall is 0.5 and AP is about 0.545, although the only real object was detected perfectly.

Expected behavior

Crowd annotations should be ignored, as in pycocotools.

Proposed fix

Skip annotations with iscrowd=1 in CocoDataset.read_annotation(), and add a test with a crowd annotation. Optionally, predictions that only match a crowd region could also be left out of the false positives, as COCO does. Happy to go with whichever you prefer.

I'd like to work on this if that's okay. Could you assign it to me?

Used an AI assistant to help check this.

Dominant language
Python
Stars
115
Forks
117
Avg merge
1d 14h
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JdeRobot/PerceptionMetrics

All issues in JdeRobot/PerceptionMetrics

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.