bug(mcp): handle malformed and partial reranker responses safely

Open Beginner friendly
#301 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

@KamilEjaz890 is already working on this.

Since Nov 18, 2026.

  • #302 by @KamilEjaz890 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start with docs-agent-mcp/mcp-server/reranker_client.py and trace how it parses reranker responses and combines scores with search hits. Check the mocked malformed JSON and partial response scenarios described in the issue. Done means failures preserve the original results, valid scored hits sort by score, unscored hits keep their relative order, and hit dictionaries are not mutated.

Written by the indexing model from the issue text.

Description

Area

mcp

What happened

The MCP reranker client does not safely handle all responses from the TEI reranking service.

There are two issues:

  1. Malformed JSON response

The response.json() call can raise a ValueError when the reranking service returns invalid JSON, such as an HTML error page.

This exception can escape the reranking function instead of gracefully falling back to the original search results.

  1. Partial reranker response

When the reranker returns scores for only some of the search hits, the current implementation combines cross-encoder scores with the original retrieval distances and sorts them together.

These values can use different score scales, such as BM25 distances and cross-encoder scores. Comparing and sorting them together can produce an incorrect ranking.

Expected behavior
  • Invalid JSON and request failures should preserve the original search results.
  • Invalid reranker entries should be ignored safely.
  • Valid scored hits should be sorted by cross-encoder score in descending order.
  • Unscored hits should retain their original relative order.
  • Original hit dictionaries should not be mutated.
  • If no valid scores are returned, the original result order should be preserved.
Reproduction

Environment

  • Repository: kubeflow/docs-agent
  • Component: MCP reranker client
  • Environment: Local development
  • Python: 3.11.9
  • Kubernetes cluster: Not required for reproducing these client-side cases.

Reproduction scenarios

Scenario 1: Malformed JSON

  1. Configure or mock the reranking HTTP request.
  2. Make the mocked reranker response contain invalid JSON, such as an HTML error page.
  3. Call the reranking function.
  4. Observe whether JSON decoding raises an exception instead of preserving the original hits.

Scenario 2: Partial reranker response

  1. Prepare multiple search hits with their original retrieval distances.
  2. Mock a reranker response that returns valid scores for only some of those hits.
  3. Call the reranking function.
  4. Observe whether scored and unscored hits are sorted together using incompatible score scales.

No live TEI service or Kubernetes cluster is required for these mocked reproduction scenarios.

Note: The issue was investigated through local code analysis. The reported test results below are from the proposed local fix; a live TEI integration reproduction has not been performed.

Suspected files

docs-agent-mcp/mcp-server/reranker_client.py

Dominant language
Python
Stars
49
Forks
122
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from kubeflow/docs-agent

All issues in kubeflow/docs-agent

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.