validate accepts unknown case-level keys, so a typo'd judges block passes with zero checks
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 50/100
Research direction
Start with the skill-up validate command and the code that decodes case and eval YAML files, then reproduce by renaming judges: to judgez: and running validate and run. #294's unknown-matcher check inside rule_based is the pattern to follow for unknown keys. Also find where a case's PASS status and grading.json totals are decided. Done means an unknown key fails validation with its name, and a case with zero executed assertions no longer reports a plain PASS. The issue leaves open whether to fail or warn, so confirm that choice with the maintainers first.
Written by the indexing model from the issue text.
Description
What happens
- A case file with a misspelled top-level key, for example
judgez:instead ofjudges:, passesskill-up validateon 0.14.0 ("eval.yaml is valid"). - The case then runs with no judge, and
skill-up runreports PASS with 0 assertions. - The same thing happened on 0.13.0 with a correctly spelled
judges:list, which that version did not support yet. The run reported PASS (100%) andgrading.jsonhad"total": 0.
Why it matters
A green run that checked nothing reads exactly like a green run that checked everything. #294 closed this for unknown matcher names inside rule_based; unknown keys at the case and eval level are still accepted.
Suggested fix
- Decode case and eval YAML strictly (unknown keys are a validation error), or at least warn with the key name.
- Report a case with zero executed assertions as something other than PASS, for example "no checks ran".
Repro
Any case file: rename judge: or judges: to judgez:, run skill-up validate, then skill-up run.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day