eval cross-judge drops the provider when splitting the judge spec
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
Research direction
Start at the eval cross-judge command and trace how its --judges value is split before the model is passed to LiteLLM. The issue identifies the failing input (provider/model) and the working double-prefix form; verify that the provider is preserved and rerun the command against an OpenAI-compatible endpoint. Done when the judge accepts the provider/model spec.
Written by the indexing model from the issue text.
Description
In eval cross-judge --judges provider/model, the spec is split on the first slash and the bare model is passed to LiteLLM without the provider, which refuses it ("LLM Provider NOT provided"). The working form is the double prefix (openai/openai/glm-5.3), which is also why the stored judge's name in arms artifacts carries it. Tested at 691c007 against an OpenAI-compatible endpoint via OPENAI_API_BASE. Everything else in the arms/judge machinery worked against that endpoint, including the judge-validity gate, which is a genuinely useful instrument.
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 167
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
Difficulty 5/5 Over a week Newbie friendliness 24/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
Maintainers usually reply within 1 day
-
User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub ActionPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 18/100
intel/dokimi#121 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
nansencenter/geodataset#39 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
awslabs/visual-asset-management-system#804 ·
Maintainers usually reply within 1 day
-
follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
odoo/odoo#297927 · 1 comment ·
Maintainers usually reply within 1 day
-
link-rot
Difficulty 1/5 Under an hour Newbie friendliness 67/100