Agent: "I also saved it to X" gets no unverified-save note when nothing was written

Open Beginner friendly
  • Is something wrong?
#7,326 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start with _WRITE_ADVERBS and _FILE_WRITE_CLAIM_PATTERNS in src/gaia/agents/base/agent.py to see why claims with “also” or “then” are missed. Run tests/unit/agents/test_answer_check_corpus.py and inspect the recorded turn for this case. Done means the missing-write answer gets the expected unverified-save note and the model is sent back to save the file.

Written by the indexing model from the issue text.

Description

bug p2

An answer that says "I also saved the report to out/report.md" when no tool wrote that file is shown to the user as is: no "I couldn't confirm all of this" note, and the model is not sent back to do the save. The same sentence without "also" is caught.

Repro

  1. New chat, any model. Ask for an edit, for example In todo.txt change milk to oat milk.
  2. Have the model finish with: Changed milk to oat milk in todo.txt. I also saved the old list to todo_old.txt. (no write_file call for todo_old.txt).

Expected: the answer ends with I couldn't confirm all of this: No successful write to todo_old.txt is recorded for this turn.

Actual: the answer is shown unchanged, so the user is told a file exists that was never written.

Phrasings that slip through today: I also saved …, I then saved …, I have also saved …, I additionally exported …. Caught today: I saved …, I just wrote …, Finally, I saved ….

🔍 Technical details

_WRITE_ADVERBS in src/gaia/agents/base/agent.py lists only just|now|already|successfully, and the first-person pattern in _FILE_WRITE_CLAIM_PATTERNS allows nothing else between I / I have and the verb. _claims_file_write("I also saved the report to out/report.md.") returns False.

Found by the recorded-turn corpus (tests/unit/agents/test_answer_check_corpus.py), which carries this turn as a strict expected failure.

Dominant language
Python
Stars
1.6k
Forks
169
Avg merge
15h 33m
Merged PRs (30d)
832

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from amd/gaia

All issues in amd/gaia

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.