Detection AUC-PR is underestimated: the curve is not anchored at recall 0, so a perfect detector scores (N-1)/N

Open
  • Is something wrong?
#672 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Siddhanthguptaa is already working on this.

Since Nov 7, 2026.

  • #673 by @Siddhanthguptaa — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
20/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
numpy, python
Domain
computer-vision, testing-qa

Research direction

Start by reading DetectionMetricsFactory.compute_auc_pr in perceptionmetrics/utils/detection_metrics.py and the tests covering detection metrics. Reproduce the single-object case from the issue and compare its AUC-PR with the expected value of 1.0. Done means perfect detections score 1.0 while curves starting with a false positive remain unchanged; an open linked pull request already addresses this issue.

Written by the indexing model from the issue text.

Description

Problem

DetectionMetricsFactory.compute_auc_pr integrates the overall precision-recall curve with the trapezoidal rule, but the curve has no point at recall = 0. The first point is already at recall = 1 / (TP + FN), so the area between recall 0 and the first detection is dropped. A detector that finds every object with no false positives therefore does not get an AUC-PR of 1: with N objects it gets (N - 1) / N, and with a single object it gets 0.

Reproduction

import numpy as np
from perceptionmetrics.utils.detection_metrics import DetectionMetricsFactory

for n in (1, 2, 5):
    m = DetectionMetricsFactory(iou_threshold=0.5)
    for i in range(n):
        box = np.array([[20 * i, 0, 20 * i + 10, 10]])
        m.update(box, [1], box, [1], [0.9 - 0.01 * i])  # prediction == ground truth
    print(n, m.compute_auc_pr(), m.get_overall_precision_recall_curve())

Output on master (4a1ae41), Python 3.11:

1 0.0 {'precision': [1.0], 'recall': [1.0]}
2 0.5 {'precision': [1.0, 1.0], 'recall': [0.5, 1.0]}
5 0.7999999970197678 {'precision': [1.0, ...], 'recall': [0.2, 0.4, 0.6, 0.8, 1.0]}

Expected 1.0 in all three cases. The value is shown in the GUI detection evaluator (AUC-PR) and is the AUC-PR entry of get_metrics_dataframe.

Proposed fix

Prepend the conventional (recall = 0, precision = 1) point before integrating, as sklearn.metrics.precision_recall_curve does, and use a stable sort so that points with equal recall keep their score order (the default sort can swap them once there are more than 16 points, which would put a lower precision before a higher one).

I compared the result against sklearn.metrics.auc over precision_recall_curve (recall rescaled for missed ground truth) on perfect detectors, curves that start with a false positive, mixed TP/FP/FN cases and 400 random detections with 20 missed objects: they agree. Curves that start with a false positive are unchanged by the fix, because the added segment has zero width there. get_overall_precision_recall_curve itself is not touched, so the GUI plot and the exported curve points stay as they are.

I'm opening a PR for this.

Dominant language
Python
Stars
114
Forks
116
Avg merge
1d 7h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JdeRobot/PerceptionMetrics

All issues in JdeRobot/PerceptionMetrics

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.