parse_json_markdown fails when a bracket appears in prose after the JSON (regression from #41959)

Start with parse_json_markdown in libs.json_in_md_parser and read test_parse_and_check_json_markdown_handles_think_fenced_and_raw_variants. Run the three reproductions and the existing multiple-block test, then verify that trailing bracketed prose no longer causes JSONDecodeError while the documented existing cases remain covered.

Bug description

parse_json_markdown fails whenever the model writes anything containing a } or ]
after the JSON — a citation such as [1], or a see [docs] line after the closing
code fence.

This is a regression from #41959 (commit 67f7a55, 2026-09-08). That change switched
extraction from fence-anchored to bracket-anchored, slicing from the first {/[ to
the last }/] in the whole string. Any bracket in trailing prose is therefore
taken to be the end of the JSON, and the resulting slice does not parse.

Before #41959 all three examples below parsed correctly.

The function is used to read LLM output in
core/rag/retrieval/dataset_retrieval.py (multi-dataset routing) and
core/llm_generator/output_parser/rule_config_generator.py, so an otherwise valid
model response is rejected.

Steps to reproduce

from libs.json_in_md_parser import parse_json_markdown

parse_json_markdown('```json\n{"a": 1}\n```\nSee [docs] for more.')
parse_json_markdown('{"action": "Final Answer", "action_input": "done"}\n\nReferences: [1]')
parse_json_markdown('Result: {"a": 1}. Note [1].')

Expected behavior

All three return the JSON object: {'a': 1}, {'action': 'Final Answer', 'action_input': 'done'}, {'a': 1}.

Actual behavior

All three raise json.decoder.JSONDecodeError:

json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 9)
json.decoder.JSONDecodeError: Extra data: line 3 column 1 (char 52)
json.decoder.JSONDecodeError: Extra data: line 1 column 9 (char 8)

Comparing the current implementation against the one immediately before #41959:

Trailing prose is clearly meant to be tolerated already: the existing test
test_parse_and_check_json_markdown_handles_think_fenced_and_raw_variants feeds
```json\n[...]\n```, error: Expecting value: line 1 column 1 (char 0) and passes —
only because that trailing text happens to contain no bracket.

Proposed fix

Decode the first complete JSON value at the opening bracket with
json.JSONDecoder().raw_decode(json_string, start_index) instead of slicing to the last
bracket. This keeps everything #41959 set out to do — anchoring on brackets, immunity to
backticks inside JSON strings and in leading prose — and makes trailing content
irrelevant regardless of what it contains.

One deliberate behavior change comes with it. test_parse_and_check_json_markdown_multiple_blocks_fails
currently pins the greedy behavior, and its own comment describes it as a limitation
rather than a requirement:

The current implementation is greedy and will match from the first opening fence to
the last closing fence, causing JSON decode failure.

Ignoring everything after the first complete JSON value necessarily means a second
fenced block is ignored too, so that case returns the first object instead of raising.
That seems strictly more useful for reading model output, but it is a semantic change,
so please say if you would rather keep it raising and I will adjust.

Environment

main at commit ad9a5be5. Python 3.12.

I have the fix and regression tests ready and will open a PR referencing this issue.