[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)

Start in src/lighteval/models/transformers/transformers_model.py, especially the generation-config update around L808-812 and the chat generation paths around L554-556 and L693-695. Reproduce the Gemma chat-generation case or inspect the existing generation flow; done means the model's declared terminators are preserved, tokenizer fallback remains available, and generation stops before max_new_tokens when the model emits its turn terminator.

Describe the bug

For generative tasks with use_chat_template=True, the transformers backend forces eos_token_id = tokenizer.eos_token_id and relies on it as the only stopping criterion. Models whose chat-turn terminator is a different token than tokenizer.eos therefore never stop: they emit their turn-end token, it is ignored, and generation runs to max_new_tokens.

Concrete case, google/gemma-4-E2B-it: tokenizer.eos_token is <eos> (id 1), but the model ends chat turns with token 106 and declares generation_config.eos_token_id = [1, 106, 50]. With a chain-of-thought MMLU task (generation_size 7168), every single generation ran to the cap: 63-95% of the returned tokens were token 106 repeated, real content was only ~400-2700 tokens.

Where it happens

src/lighteval/models/transformers/transformers_model.py (current main):

• L808-812: generation_config.update(..., eos_token_id=self.tokenizer.eos_token_id, ...) clobbers whatever eos_token_id the model's own generation config declares (the dict copied at L808 is overwritten by the update).
• L554-556 and L693-695: both generation paths assume "for chat models, generation stops with EOS token" (stop_tokens = [] / [self.tokenizer.eos_token]), which bakes in the same assumption. This dates back to #115.

Task-level stop_sequence cannot work around it for chat models, since it is dropped on that path.

Measured impact (Gemma-4-E2B-it, MMLU CoT, one item, RTX PRO 6000, bf16)

Besides ~3x wall-clock waste, this silently corrupts tokens_generated as a measurement (it tracks the cap, not the model), which matters for any analysis using generation length.

Suggested fix (validated above)

Prefer the model's declared terminators, falling back to the tokenizer's:

eos_token_id=(self.model.generation_config.eos_token_id or self.tokenizer.eos_token_id),

Happy to open a PR with this change.

Related

Same footgun class (Gemma turn terminator != <eos>) as huggingface/transformers#38182 and unslothai/unsloth#5386, but this instance is in lighteval's own generate call.

Version

Reproduced against lighteval main (verified still present at HEAD as of 2026-07-02; originally found at 3fd1526, 2026-06-26).