codalab/codabench

Codabench is a flexible, easy-to-use and reproducible benchmarking platform. Check our paper: https://hubs.li/Q01fwRWB0

Stars: 176
Forks: 74
Open beginner issues: 2
Indexed issues: 103
Avg merge: 5d 3h
Merged PRs (30d): 21

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 20, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

register error for 500 Open

Difficulty 4/5 3-5 days Newbie friendliness 15/100

codalab/codabench#2559 · 1 comment · Opened Sep 21, 2026
Invisible submissions stuck on scoreboard unevaluated Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

codalab/codabench#2558 · Opened Sep 21, 2026
Intermittent long-running submissions eventually fail Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

codalab/codabench#2557 · Opened Sep 21, 2026
Displaying incorrect number of participants Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

codalab/codabench#2554 · Opened Sep 18, 2026
MOT17 / MOT20 submissions stuck in Running for ~1 hour, then fail with "Child task failed or non-zero return code" and no logs Open

Difficulty 4/5 3-5 days Newbie friendliness 40/100

codalab/codabench#2548 · Opened Sep 16, 2026
[Urgent] Organizer accidentally deleted a leaderboard submission, can it be restored from backup? Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

codalab/codabench#2547 · Opened Sep 16, 2026
API Throttling - re-implement and fix for compute workers Open

codalab/codabench#2545 · 1 comment · 1 assignee · Opened Sep 16, 2026
CircleCI tests: use the `site_worker` Open

Post-it

codalab/codabench#2541 · 2 assignees · Opened Sep 15, 2026
Bulk download results/submissions: dead code in `submissions.py` Open

Post-it

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

codalab/codabench#2538 · Opened Sep 14, 2026
Error is not clear after reaching the computation limit of a submission Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

codalab/codabench#2529 · Opened Sep 7, 2026
MOT20 leaderboard ranking issue Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

codalab/codabench#2512 · Opened Aug 23, 2026
No logs, but the submission failed... Open

Bug Competition-specific

Difficulty 4/5 3-5 days Newbie friendliness 40/100

codalab/codabench#2506 · 3 comments · Opened Aug 20, 2026
Clear leaderboard Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

codalab/codabench#2505 · 1 comment · Opened Aug 18, 2026
Human in the Loop Issues Open

Enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

codalab/codabench#2504 · Opened Aug 18, 2026
Public Competitions API - next and previous links Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

codalab/codabench#2503 · Opened Aug 17, 2026
Submissions API architecture Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

codalab/codabench#2497 · Opened Aug 8, 2026
Dump issue: dates are saved with exact time Open

Beginner Bug

Difficulty 3/5 1-2 days Newbie friendliness 62/100

codalab/codabench#2490 · Opened Aug 5, 2026
Site Worker not cleaning up after itself (competition bundle) Open

Bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

codalab/codabench#2478 · Opened Jul 24, 2026
MOT20 leaderboard ranks a submission with no evaluation scores as #1 Open

Difficulty 3/5 1-2 days Newbie friendliness 52/100

codalab/codabench#2470 · Opened Jul 23, 2026
Task 2 repeatedly fails on ingestion_hostname-worker2 with no logs Open

Competition-specific User-specific

Difficulty 4/5 3-5 days Newbie friendliness 35/100

codalab/codabench#2465 · 1 comment · Opened Jul 18, 2026
MinIO has been discontinued Open

P1 Setup

Difficulty 4/5 3-5 days Newbie friendliness 45/100

codalab/codabench#2459 · Opened Jul 3, 2026
Username problem in email "User applied to your competition" Open

Beginner Bug

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

codalab/codabench#2445 · Opened Jun 24, 2026
Codabench package Open

Post-it

Difficulty 5/5 Over a week Newbie friendliness 25/100

codalab/codabench#2440 · Opened Jun 23, 2026
Worker monitoring feature: needed improvements Open

Enhancement

codalab/codabench#2438 · 1 assignee · Opened Jun 23, 2026
compute_worker re-executes submissions on broker redelivery, causing duplicate scores, hostname overwrite, and status flipping Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

codalab/codabench#2433 · Opened Jun 18, 2026
Improve "Participants" table Open

Enhancement UI

Difficulty 3/5 1-2 days Newbie friendliness 68/100

codalab/codabench#2428 · Opened Jun 18, 2026
Participant routing feature: needed improvements Open

Enhancement

codalab/codabench#2427 · 1 comment · 1 assignee · Opened Jun 18, 2026
Lost submissions on client side: POST /api/submissions/ is not idempotent Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

codalab/codabench#2425 · Opened Jun 17, 2026
Data integrity: score or leaderboard write fails after Finished status Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

codalab/codabench#2423 · Opened Jun 17, 2026
Submissions stuck in "Scoring" when broker error occurs during compute_worker PATCH (non-transactional re-enqueue) Open

Workers / Queues

Difficulty 4/5 3-5 days Newbie friendliness 48/100

codalab/codabench#2419 · Opened Jun 16, 2026
_update_submission() has no retry and _update_status() silently swallows exceptions, causing permanent submission loss Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

codalab/codabench#2415 · Opened Jun 16, 2026
Submission cleanup only recovers Running submissions, not Submitted/Preparing/Scoring Open

Difficulty 3/5 1-2 days Newbie friendliness 75/100

codalab/codabench#2413 · Opened Jun 16, 2026
Renaming "administrator", "collaborator", "organizers", etc. Open

Documentation Post-it

Difficulty 4/5 3-5 days Newbie friendliness 35/100

codalab/codabench#2412 · 1 comment · Opened Jun 16, 2026
Registration failed with 500 error and activation email not received Open

Bug User-specific

Difficulty 4/5 3-5 days Newbie friendliness 25/100

codalab/codabench#2406 · 1 comment · Opened Jun 8, 2026
Add proper configuration management for compute-worker Open

Enhancement In Progress python python:uv Setup Workers / Queues

codalab/codabench#2404 · 1 assignee · Opened Jun 4, 2026
GPU worker: No score output Open

Competition-specific Workers / Queues

codalab/codabench#2399 · 16 comments · 1 assignee · Opened Jun 1, 2026
MOT20 submission failed: child task failed Open

Bug Competition-specific

Difficulty 4/5 3-5 days Newbie friendliness 25/100

codalab/codabench#2384 · 1 comment · Opened May 24, 2026
Compute worker: changing `HOST_DIRECTORY` not working Open

Bug Workers / Queues

Difficulty 3/5 1-2 days Newbie friendliness 48/100

codalab/codabench#2379 · Opened May 19, 2026
Feedback from a competition: auto-migration bug, stuck submissions, and username reset on re-run Open

Bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

codalab/codabench#2375 · 1 comment · Opened May 13, 2026
analytics page misreporting Open

Bug

codalab/codabench#2354 · 2 comments · 1 assignee · Opened Apr 28, 2026
[COCO Segmentation] test-dev 2017 evaluation sever unavailable and Codabench do not have COCO benchmark Open

Competition-specific Discussion

Difficulty 5/5 Over a week Newbie friendliness 25/100

codalab/codabench#2352 · 1 comment · Opened Apr 26, 2026
Method to track inactive/stale users and competitions Open

Enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

codalab/codabench#2344 · 1 comment · Opened Apr 20, 2026
MOT20 failed submission Open

Bug Competition-specific Workers / Queues

codalab/codabench#2343 · 3 comments · 1 assignee · Opened Apr 19, 2026
Resource Interface: Submission file size display problem Open

Beginner Bug UI

Difficulty 3/5 1-2 days Newbie friendliness 52/100

codalab/codabench#2335 · Opened Apr 15, 2026
Resource Interface: Improve time format Open

Beginner Enhancement UI

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

codalab/codabench#2334 · Opened Apr 15, 2026
Sorting columns in Resources interface is flawed Open

Bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

codalab/codabench#2327 · Opened Apr 9, 2026
Computer worker upgrade feature Open

Enhancement Post-it

Difficulty 4/5 3-5 days Newbie friendliness 25/100

codalab/codabench#2320 · Opened Apr 7, 2026
Codabench Account Activation Email Not Received Open

P1 Production

Difficulty 4/5 3-5 days Newbie friendliness 35/100

codalab/codabench#2263 · 8 comments · Opened Mar 13, 2026
More control over submissions for organizers Open

Enhancement P1

Difficulty 5/5 Over a week Newbie friendliness 25/100

codalab/codabench#2249 · 1 comment · Opened Mar 9, 2026
Auto-migration not working for multi-task Open

Bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

codalab/codabench#2247 · 1 comment · Opened Mar 9, 2026

Showing the newest 100

This page lists what was indexed most recently. Advanced filter has the whole inventory, narrowed by language, difficulty and how long a task takes.

Open advanced filter

codalab/codabench

Showing the newest 100

Get new issues in your inbox