validate accepts unknown case-level keys, so a typo'd judges block passes with zero checks

Open
#365 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
50/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, json, yaml
Domain
cli, devtools

Research direction

Start with the skill-up validate command and the code that decodes case and eval YAML files, then reproduce by renaming judges: to judgez: and running validate and run. #294's unknown-matcher check inside rule_based is the pattern to follow for unknown keys. Also find where a case's PASS status and grading.json totals are decided. Done means an unknown key fails validation with its name, and a case with zero executed assertions no longer reports a plain PASS. The issue leaves open whether to fail or warn, so confirm that choice with the maintainers first.

Written by the indexing model from the issue text.

Description

What happens

  1. A case file with a misspelled top-level key, for example judgez: instead of judges:, passes skill-up validate on 0.14.0 ("eval.yaml is valid").
  2. The case then runs with no judge, and skill-up run reports PASS with 0 assertions.
  3. The same thing happened on 0.13.0 with a correctly spelled judges: list, which that version did not support yet. The run reported PASS (100%) and grading.json had "total": 0.

Why it matters
A green run that checked nothing reads exactly like a green run that checked everything. #294 closed this for unknown matcher names inside rule_based; unknown keys at the case and eval level are still accepted.

Suggested fix

  1. Decode case and eval YAML strictly (unknown keys are a validation error), or at least warn with the key name.
  2. Report a case with zero executed assertions as something other than PASS, for example "no checks ran".

Repro
Any case file: rename judge: or judges: to judgez:, run skill-up validate, then skill-up run.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.