miniswen: record the served model version on each turn
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 70/100
Research direction
Start with Miniswen::Agent#payload and trace how its response data is written into the trajectory and ATIF step metadata. Read the existing tests for those outputs, then add coverage for recording response.model as the served model. Done when both trajectory formats include it and tests confirm the value.
Written by the indexing model from the issue text.
Description
Trajectories record the requested model (openrouter/...#effort) and, since 1.6.0, the OpenRouter upstream provider of each turn, but not the model version the provider says actually answered. Benchmark results can therefore silently mix models.
We hit this running gemini/gemini-4-hydrogen#high directly against the Gemini API with lemans 1.5.2 (ruby_llm 1.16): every response reported modelVersion: "3.1-flash-lite-05-2026", not gemini-4-hydrogen. Bisecting the first-turn request showed the trigger: ruby_llm 1.16 sends the :system message as a leading "user" turn. The identical request with the system prompt in systemInstruction was answered as gemini-4-hydrogen; same prompt, same endpoint and billing. Others report the same relabelling on the Gemini API (Google AI forum threads 199710, 199771, 180734).
lemans 1.6.0 (ruby_llm 2.1.1) sends systemInstruction, and we no longer see the mismatch: on 1.6.0 every settled request of our gemini-4-carbon and gemini-4-neon runs reports exactly that model. Our 1.5.2 Hydrogen run, with the system prompt moved to systemInstruction by hand, reports gemini-4-hydrogen on all but the 7 requests sent before that fix. Still, nothing in a trajectory would reveal a substitution like this one, on any provider.
ruby_llm 2.1.1 already parses it: RubyLLM::Message#model ("the ID of the model that produced the message") is Gemini's modelVersion and the model field of OpenAI (Responses and Chat Completions) and Anthropic responses. Proposal: add it to Miniswen::Agent#payload, e.g. served_model: response.model, next to provider, and to the ATIF step metadata, so reports can flag turns where the served model differs from the requested one.
Happy to send the PR (payload + trajectory + a test) if this direction works for you.
- Dominant language
- Ruby
- Stars
- 99
- Forks
- 11
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 2
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rails/lemans
-
Secrets filter leaves the tail of a secret when another secret is its prefixPossibly taken @RaphaelFakhri claimed this 17 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
This is great.Open
Difficulty 1/5 Under an hour Newbie friendliness 10/100
Similar issues
-
開発用DockerイメージにPlaywrightのシステム依存を追加するPossibly taken @momocus claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
sakazuki-org/sakazuki#1453 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
jenkins-infra/jenkins.io#9615 · 1 comment ·
Maintainers usually reply within 1 day
-
fix(ocr): compile the Tesseract-free pipeline with Tokio runtimePossibly taken @Goldziher claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Metric tooltips that contradict how the metric is calculatedPossibly taken @aniadeadu claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
WikiEducationFoundation/WikiEduDashboard#7303 · 1 comment ·
Maintainers usually reply within 1 day
-
WhatsApp Cloud template sync saves a partial template list when one page of the Graph API response failsPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days