RAG: _attempt_answer crashes on unparseable model output

Open Beginner friendly
#52 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
88/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in docling_agent/agent/rag.py with _attempt_answer and compare its find_json_dicts handling with _select_section. Add a test that feeds unparseable output through a mock session and verify the RAG loop continues with a cannot-answer attempt rather than crashing. Done means junk output no longer raises IndexError or KeyError and the failed iteration remains visible in the trace.

Written by the indexing model from the issue text.

Description

_attempt_answer in docling_agent/agent/rag.py parses the model response without any check:

d = find_json_dicts(answer)[0]
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])

_select_section, in the same file, guards the exact same pattern and falls back to
picking the first unvisited section. The other find_json_dicts call sites
(orchestrator, enricher, editor) all check the result too. This is the only
unguarded one.

It can trigger on every backend. On the mellea backend the retry budget is best
effort: when rejection sampling runs out, the session returns a failed sample
instead of raising. On the direct HTTP backends (Ollama and the OpenAI-compatible
ones), requirements are not enforced at all, so the first response without a valid
JSON block reaches the parse directly.
Either way this line raises IndexError (no JSON found) or KeyError (missing key)
and the whole RAG run aborts. Without the crash, _rag_loop would have moved on to
the next section, or returned the partial answer it already had.

Fix, mirroring _select_section:

dicts = find_json_dicts(answer)
d = dicts[0] if dicts else {}
if not isinstance(d.get("can_answer"), bool) or not isinstance(d.get("response"), str):
    log_warning(f"Unparseable answer attempt for section {section_ref!r}; treating as 'cannot answer'.")
    return AnswerAttempt(can_answer=False, response="Could not extract a usable answer from this section.")
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])

can_answer=False just sends the loop to the next section, and the failed iteration
stays visible in the trace.

About 6 lines plus a test that feeds junk through a mock session. Happy to open the PR.

PS: found this while working on #25.

Dominant language
Python
Stars
167
Forks
21
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from docling-project/docling-agent

All issues in docling-project/docling-agent

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.