[BUG]: loo_eval.py always asks the conversation models about 3I/ATLAS and ignores the prompts you pass

Aperta
#338 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@arshil42 ci sta già lavorando.

Dal 18/11/2026.

  • #339 di @arshil42 — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Bug Description

simple/loo_eval.py takes --prompt for the greedy completions, but only the base models use it. The conversation models are always asked the same hardcoded question:

result[name] = {"completions": {q: complete(model, tokenizer, "What is 3I/ATLAS?", chat=True) for q in ["chat"]}}

simple/README.md describes loo_eval.py as the general comparison of an original run with any fork that removed documents (ovl.py fork --exclude-page ..., then loo_eval.py). For a fork that removed other pages, the base models answer the prompts you pass, but the chat completions of both conversation models still ask about 3I/ATLAS, and there is no option to change that.

Steps to Reproduce
  1. Fork a run that removes a page other than the 3I/ATLAS ones, train it, and run python simple/loo_eval.py --data PREP --original RUN --fork FORK --prompt "Hamlet is".
  2. original_base and fork_base complete "Hamlet is", but original_conversation and fork_conversation hold the answers to "What is 3I/ATLAS?".
Expected Behavior

The question asked to the conversation models can be set, like --prompt for the base models, and defaults to the current "What is 3I/ATLAS?" so the output of existing commands does not change.

Impact

A leave-one-out comparison of another removed page has no chat evidence about that page.

I'd like to fix this with a --chat-prompt option (default "What is 3I/ATLAS?", the output key stays chat), a test, and a note in simple/README.md. This was found with an AI coding agent (Claude Code, Claude Opus 5.5), and I checked it locally.

Environment Details

main at e04d797, Windows, Python 3.12, CPU.

Lingua principale
Python
Stelle
26
Fork
64
Merge medio
2g 11h
PR unite (30g)
71

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di AOSSIE-Org/OpenVerifiableLLM

Tutte le issue di AOSSIE-Org/OpenVerifiableLLM

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.