Re-run and re-submit features
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
Research direction
This is a tracking issue, not a single task, and the payload names no files or tests. Start by reading the linked pull requests (#2620, #2642, #2644) and related issues (#2636, #2637, #2655) to see what is already under way. The open items are an edited-task flag, in-place re-running, and the re-submit feature, so a newcomer should first ask the maintainers which one is free to pick up.
Written by the indexing model from the issue text.
Description
I create this issue to clarify and track the development around the "re-run" and the "re-submit" features.
TODO
Flagging mixed leaderboard
- Removed task: https://github.com/codalab/codabench/pull/2642
- Edited task: ??? (PR to make)
Related issue: #2637
Rework "re-running" feature
- Error/warning if task removed/edited: https://github.com/codalab/codabench/pull/2620
- In-place re-running: (PR to make)
- Codabench erases all data related to the submission then launches it again (same ID, re-run will not make a new submission. Quota is not consumed for the participant. Old tasks/datasets are used since it's re-running in the same state)
-
NO (behavior change on re-run) https://github.com/codalab/codabench/pull/2624
Add "re-submit" feature
Other "re-run" related issues
- #2655
- #2636
- Dominant language
- Python
- Stars
- 183
- Forks
- 80
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 35
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from codalab/codabench
-
Wrong count for failed submissions in Resource interfacePossibly taken @ihsaan-ullah claimed this 6 days ago. OpenBug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Resource Interface: Improve time formatPossibly taken @ihsaan-ullah claimed this 5 days ago. OpenBeginner Enhancement UI
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Revert "Run E2E and Unit tests in parallel"Possibly taken @ObadaS claimed this 1 day ago. OpenPost-it
codalab/codabench#2658 · 1 assignee ·
Maintainers usually reply within 1 day
-
Bug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
External-instance
Difficulty 4/5 3-5 days Newbie friendliness 12/100
Maintainers usually reply within 1 day
All issues in codalab/codabench
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Deltares/imod-python#1940 ·
Maintainers usually reply within 1 day
-
effort: low prio: low ready
Difficulty 1/5 Under an hour Newbie friendliness 82/100
digitalfabrik/integreat-cms#4621 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
ioos/compliance-checker#1375 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day