RAG: _attempt_answer crashes on unparseable model output
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
Research direction
Start in docling_agent/agent/rag.py with _attempt_answer and compare its find_json_dicts handling with _select_section. Add a test that feeds unparseable output through a mock session and verify the RAG loop continues with a cannot-answer attempt rather than crashing. Done means junk output no longer raises IndexError or KeyError and the failed iteration remains visible in the trace.
Written by the indexing model from the issue text.
Description
_attempt_answer in docling_agent/agent/rag.py parses the model response without any check:
d = find_json_dicts(answer)[0]
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
_select_section, in the same file, guards the exact same pattern and falls back to
picking the first unvisited section. The other find_json_dicts call sites
(orchestrator, enricher, editor) all check the result too. This is the only
unguarded one.
It can trigger on every backend. On the mellea backend the retry budget is best
effort: when rejection sampling runs out, the session returns a failed sample
instead of raising. On the direct HTTP backends (Ollama and the OpenAI-compatible
ones), requirements are not enforced at all, so the first response without a valid
JSON block reaches the parse directly.
Either way this line raises IndexError (no JSON found) or KeyError (missing key)
and the whole RAG run aborts. Without the crash, _rag_loop would have moved on to
the next section, or returned the partial answer it already had.
Fix, mirroring _select_section:
dicts = find_json_dicts(answer)
d = dicts[0] if dicts else {}
if not isinstance(d.get("can_answer"), bool) or not isinstance(d.get("response"), str):
log_warning(f"Unparseable answer attempt for section {section_ref!r}; treating as 'cannot answer'.")
return AnswerAttempt(can_answer=False, response="Could not extract a usable answer from this section.")
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
can_answer=False just sends the loop to the next section, and the failed iteration
stays visible in the trace.
About 6 lines plus a test that feeds junk through a mock session. Happy to open the PR.
PS: found this while working on #25.
- Dominant language
- Python
- Stars
- 167
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from docling-project/docling-agent
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
docling-project/docling-agent#25 · 6 comments · 2 assignees ·
All issues in docling-project/docling-agent
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100