miniswen: record the served model version on each turn

Open
#23 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
70/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
ruby
Domain
backend

Research direction

Start with Miniswen::Agent#payload and trace how its response data is written into the trajectory and ATIF step metadata. Read the existing tests for those outputs, then add coverage for recording response.model as the served model. Done when both trajectory formats include it and tests confirm the value.

Written by the indexing model from the issue text.

Description

Trajectories record the requested model (openrouter/...#effort) and, since 1.6.0, the OpenRouter upstream provider of each turn, but not the model version the provider says actually answered. Benchmark results can therefore silently mix models.

We hit this running gemini/gemini-4-hydrogen#high directly against the Gemini API with lemans 1.5.2 (ruby_llm 1.16): every response reported modelVersion: "3.1-flash-lite-05-2026", not gemini-4-hydrogen. Bisecting the first-turn request showed the trigger: ruby_llm 1.16 sends the :system message as a leading "user" turn. The identical request with the system prompt in systemInstruction was answered as gemini-4-hydrogen; same prompt, same endpoint and billing. Others report the same relabelling on the Gemini API (Google AI forum threads 199710, 199771, 180734).

lemans 1.6.0 (ruby_llm 2.1.1) sends systemInstruction, and we no longer see the mismatch: on 1.6.0 every settled request of our gemini-4-carbon and gemini-4-neon runs reports exactly that model. Our 1.5.2 Hydrogen run, with the system prompt moved to systemInstruction by hand, reports gemini-4-hydrogen on all but the 7 requests sent before that fix. Still, nothing in a trajectory would reveal a substitution like this one, on any provider.

ruby_llm 2.1.1 already parses it: RubyLLM::Message#model ("the ID of the model that produced the message") is Gemini's modelVersion and the model field of OpenAI (Responses and Chat Completions) and Anthropic responses. Proposal: add it to Miniswen::Agent#payload, e.g. served_model: response.model, next to provider, and to the ATIF step metadata, so reports can flag turns where the served model differs from the requested one.

Happy to send the PR (payload + trajectory + a test) if this direction works for you.

Dominant language
Ruby
Stars
99
Forks
11
Avg merge
3d 9h
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rails/lemans

All issues in rails/lemans

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.