Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

TAPVid-3D: per_trajectory and local_neighborhood scaling always score 0 in evaluate_model.py

Open Beginner friendly
  • Is something wrong?
#173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
numpy, python
Domain
computer-vision, testing-qa

Research direction

Start in tapnet/tapvid3d/evaluation/metrics.py at compute_tapvid3d_metrics, focusing on how the batch axis is added for unbatched tracks and how query_points is indexed by np.take_along_axis. Run the issue's perfect-prediction snippet to reproduce the ValueError for per_trajectory and the AxisError for local_neighborhood, then check the np.stack path when get_trackwise_metrics=False. Done means the snippet reports average_jaccard 1 for all three scalings and existing scaling/order combinations keep identical outputs.

Written by the indexing model from the issue text.

Description

With --depth_scalings=per_trajectory or local_neighborhood, evaluate_model.py reports 0 for every metric (3D-AJ, APD and OA). compute_tapvid3d_metrics raises for both scalings when it is called the way evaluate_model.py calls it (unbatched, order='t n'), and for each video the exception is logged and replaced with ZERO_METRICS_DICT (evaluate_model.py#L236-L248). These are the scalings behind Tables 5 and 6 of the TAPVid-3D paper. The default median scaling is not affected.

The two causes, both in metrics.py:

  • per_trajectory: unbatched tracks get a batch axis ('() n t', L183-L191) but query_points does not, so np.take_along_axis (L204-L213) gets [n, 1] indices for a [1, n, t] array and raises ValueError.
  • local_neighborhood: only unbatched input is allowed (L177-L181), so with get_trackwise_metrics=False each per-threshold metric is 0-d and np.stack(..., axis=-2) (L319-L326) raises AxisError.

Perfect-prediction check (from a checkout of the repo, only needs numpy and einops):

import numpy as np
from tapnet.tapvid3d.evaluation import metrics

rng = np.random.default_rng(0)
num_frames, num_points = 10, 4
tracks = rng.uniform(1.0, 5.0, size=(num_frames, num_points, 3))  # 't n' order, as in evaluate_model.py
occluded = np.zeros((num_frames, num_points), dtype=bool)
query_points = np.zeros((num_points, 3))  # (t, y, x), all queried on frame 0
intrinsics = np.array([100.0, 100.0, 50.0, 50.0])  # fx, fy, cx, cy

for scaling in ['median', 'per_trajectory', 'local_neighborhood']:
  try:
    out = metrics.compute_tapvid3d_metrics(
        gt_occluded=occluded, gt_tracks=tracks,
        pred_occluded=occluded, pred_tracks=tracks,  # perfect prediction
        intrinsics_params=intrinsics, scaling=scaling,
        query_points=query_points, order='t n')
    print(f"{scaling}: average_jaccard = {out['average_jaccard']}")
  except Exception as e:
    print(f'{scaling}: {type(e).__name__}: {e}')

Output at 730cda1:

median: average_jaccard = [1.]
per_trajectory: ValueError: `indices` and `arr` must have the same number of dimensions
local_neighborhood: AxisError: axis -2 is out of bounds for array of dimension 1

A possible fix:

--- a/tapnet/tapvid3d/evaluation/metrics.py
+++ b/tapnet/tapvid3d/evaluation/metrics.py
@@ -189,6 +189,8 @@ def compute_tapvid3d_metrics(
   pred_occluded = einops.rearrange(pred_occluded, f'{order} -> {output_order}')
   gt_tracks = einops.rearrange(gt_tracks, f'{order} d -> {output_order} d')
   pred_tracks = einops.rearrange(pred_tracks, f'{order} d -> {output_order} d')
+  if output_order == '() n t' and np.ndim(query_points) == 2:
+    query_points = query_points[np.newaxis]  # Same batch axis as the tracks.
 
   summing_axis = (-1,) if get_trackwise_metrics else (-2, -1)
   evaluation_weights = np.ones(gt_occluded.shape)
@@ -221,6 +223,14 @@ def compute_tapvid3d_metrics(
               thresh=local_neighborhood_thresh,
           )
       )
+      if not get_trackwise_metrics:
+        # Add a batch axis, so that each metric has shape [1] as with the other
+        # scalings (and as in ZERO_METRICS_DICT in evaluate_model.py).
+        gt_occluded = gt_occluded[np.newaxis]
+        gt_tracks = gt_tracks[np.newaxis]
+        pred_occluded = pred_occluded[np.newaxis]
+        pred_tracks = pred_tracks[np.newaxis]
+        evaluation_weights = evaluation_weights[np.newaxis]
   else:
     either_occluded = np.logical_or(gt_occluded, pred_occluded)
     nan_mat = np.full(pred_norms.shape, np.nan)

With it, the check above gives an average_jaccard of 1 for all three scalings. local_neighborhood then returns metrics of shape [1], like the other scalings and ZERO_METRICS_DICT, so a video that fails (for example, because its prediction file is missing) still averages in as 0. Over every combination of scaling, order ('n t', 't n', 'b n t', 'b t n'), get_trackwise_metrics and use_fixed_metric_threshold, everything that already ran before the change gives identical outputs (same shapes and values).

On the released SpatialTracker predictions for DriveTrack minival (the GCS folder used in run_evaluate_model.sh), median gives 5.84 / 10.25 / 81.95 (3D-AJ / APD / OA), which matches the DriveTrack entry of the SpatialTracker* (minival) row of Table 3: 5.8 / 10.2 / 82.0. With the fix, per_trajectory gives 8.61 / 15.13 / 81.95 and local_neighborhood 7.85 / 13.91 / 81.94. Tables 5 and 6 only report full_eval, so these minival numbers are not directly comparable to them.

This isn't my job, just something I enjoy: checking how benchmarks are actually scored.

Dominant language
Jupyter Notebook
Stars
2.2k
Forks
215
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from google-deepmind/tapnet

All issues in google-deepmind/tapnet

Similar issues

More Computer Vision issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.