bug(mcp): handle malformed and partial reranker responses safely
Maintainers usually reply within 4 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 85/100
Research direction
Start with docs-agent-mcp/mcp-server/reranker_client.py and trace how it parses reranker responses and combines scores with search hits. Check the mocked malformed JSON and partial response scenarios described in the issue. Done means failures preserve the original results, valid scored hits sort by score, unscored hits keep their relative order, and hit dictionaries are not mutated.
Written by the indexing model from the issue text.
Description
Area
mcp
What happened
The MCP reranker client does not safely handle all responses from the TEI reranking service.
There are two issues:
- Malformed JSON response
The response.json() call can raise a ValueError when the reranking service returns invalid JSON, such as an HTML error page.
This exception can escape the reranking function instead of gracefully falling back to the original search results.
- Partial reranker response
When the reranker returns scores for only some of the search hits, the current implementation combines cross-encoder scores with the original retrieval distances and sorts them together.
These values can use different score scales, such as BM25 distances and cross-encoder scores. Comparing and sorting them together can produce an incorrect ranking.
Expected behavior
- Invalid JSON and request failures should preserve the original search results.
- Invalid reranker entries should be ignored safely.
- Valid scored hits should be sorted by cross-encoder score in descending order.
- Unscored hits should retain their original relative order.
- Original hit dictionaries should not be mutated.
- If no valid scores are returned, the original result order should be preserved.
Reproduction
Environment
- Repository:
kubeflow/docs-agent - Component: MCP reranker client
- Environment: Local development
- Python: 3.11.9
- Kubernetes cluster: Not required for reproducing these client-side cases.
Reproduction scenarios
Scenario 1: Malformed JSON
- Configure or mock the reranking HTTP request.
- Make the mocked reranker response contain invalid JSON, such as an HTML error page.
- Call the reranking function.
- Observe whether JSON decoding raises an exception instead of preserving the original hits.
Scenario 2: Partial reranker response
- Prepare multiple search hits with their original retrieval distances.
- Mock a reranker response that returns valid scores for only some of those hits.
- Call the reranking function.
- Observe whether scored and unscored hits are sorted together using incompatible score scales.
No live TEI service or Kubernetes cluster is required for these mocked reproduction scenarios.
Note: The issue was investigated through local code analysis. The reported test results below are from the proposed local fix; a live TEI integration reproduction has not been performed.
Suspected files
docs-agent-mcp/mcp-server/reranker_client.py
- Dominant language
- Python
- Stars
- 49
- Forks
- 122
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from kubeflow/docs-agent
-
bug(pipelines): shortcode stripping deletes text between a {{< >}} shortcode and the next {{% %}}Possibly taken @Ashborne-lab claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
kubeflow/docs-agent#293 ·
Maintainers usually reply within 4 days
-
bug(mcp): release-date version rerank matches 1.1 against 1.10, v2.1.1 and 11.1Possibly taken @Aryan-Pardeshi claimed this 23 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
kubeflow/docs-agent#279 ·
Maintainers usually reply within 4 days
-
bug(mcp): Milvus search failures escape the tool boundaryPossibly taken @sanjayy0612 claimed this 66 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
kubeflow/docs-agent#251 ·
Maintainers usually reply within 4 days
-
bug(frontend): the chat widget calls two functions that no longer existPossibly taken @Neilblaze claimed this 71 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
kubeflow/docs-agent#247 ·
Maintainers usually reply within 4 days
-
fix: update API server paths in README to point to legacy directoryPossibly taken @pratik-naik003 claimed this 105 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 92/100
kubeflow/docs-agent#225 · 1 reaction ·
Maintainers usually reply within 4 days
All issues in kubeflow/docs-agent
Similar issues
-
Update the demo docsOpen
Difficulty 1/5 Under an hour Newbie friendliness 72/100
pulp/pulp_workflow#78 ·
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
pgadmin-org/pgadmin4#10797 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
gcivil-nyu-org/team5-mon-fall26#132 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 76/100
gcivil-nyu-org/team4-mon-fall26#291 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
gcivil-nyu-org/team3-mon-fall26#147 ·
Maintainers usually reply within 2 days