Gateway loses model-request events for malformed or non-object upstream JSON
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 20/100
Research direction
Read agentlightning.server.proxy.forward_request and the existing endpoint cases in tests/server/test_proxy_response.py and tests/server/test_endpoints.py. Start by reproducing the malformed JSON and non-object response cases using the offline setup described in the issue. Done means these responses return a gateway error, record a model-request event, preserve upstream HTTP error statuses, and release the in-flight counter; linked PR #658 is already open.
Written by the indexing model from the issue text.
Description
Problem
On main at 05c7fe133175653e48c990546a3509e52dc56512, forward_request assumes the upstream response is a JSON object:
- malformed JSON with
Content-Type: application/jsonraises duringresponse.json(); - JSON arrays, strings, numbers and
nullreach_pop_routed_expertsand fail before_capture_event; - a non-JSON HTTP 200 response becomes
{}and is recorded as a successful model request.
The first two paths lose the model-request event, making a failed rollout harder to diagnose.
Offline reproduction
Replace agentlightning.server.proxy._send_upstream_with_retries with an async function returning httpx.Response(200, text="null", headers={"content-type": "application/json"}). Register a rollout and model, then POST a normal chat-completion request through the gateway. The request raises instead of returning a gateway error, and no model-request event is recorded. A response containing { reproduces the JSON parsing failure; HTTP 200 with HTML reproduces the empty successful response.
Proposed scope
I have a focused local patch and eight endpoint regression cases. The patch returns a structured upstream-response error for malformed/non-object/non-JSON responses, maps invalid responses below HTTP 400 to 502, and preserves existing upstream HTTP error statuses. It records the error and retry count and releases the in-flight counter. Valid object responses keep their current path.
Would this error/status contract be appropriate for a PR?
Validation
Windows, Python 3.12.14; FastAPI 0.162.0, HTTPX 0.28.1, pytest 9.1.1. Dependencies were installed from the declared ranges, not the frozen lockfile. No model calls or GPU use.
- Eight new cases: 8 failed before the patch.
python -m pytest -q tests/server/test_proxy_response.py tests/server/test_endpoints.py tests/test_client.py --tb=short: 51 passed after the patch.- Server/controller/examples/client tests: patched 101 passed, 9 failed; unchanged upstream 93 passed, the same 9 failed. These failures reference POSIX
os.killpg, unavailable on Windows. - Ruff lint/format and
git diff --check: passed.
I searched open PRs for upstream JSON response handling and non-JSON handling. #587 concerns request bodies; #574 concerns stored triplet export. Neither handles this response-ingestion path. AI assistance was used for investigation, the local patch and test execution.
- Dominant language
- Python
- Stars
- 18.8k
- Forks
- 1.7k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/agent-lightning
-
Low-precision rollout normalization can overflow its divisor and erase advantagesPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
microsoft/agent-lightning#661 ·
Maintainers usually reply within 2 days
-
RayPPOTrainer._dump_generations() missing gts argument in _train_step callPossibly taken @yunkai-yyk claimed this 253 days ago. Openbug v0.x verl
Difficulty 1/5 Under an hour Newbie friendliness 68/100
microsoft/agent-lightning#492 ·
Maintainers usually reply within 2 days
-
enhancement store v0.x
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
microsoft/agent-lightning#372 · 10 comments ·
Maintainers usually reply within 2 days
-
Terminal rollout projection drops numeric dataset ID zeroPossibly taken @notoufa claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
microsoft/agent-lightning#659 ·
Maintainers usually reply within 2 days
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
microsoft/agent-lightning#650 · 1 comment ·
Maintainers usually reply within 2 days
All issues in microsoft/agent-lightning
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ComposioHQ/awesome-claude-skills#2398 ·
Maintainers usually reply within 2 days
-
bug converter
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
grafana/sigma-rule-deployment#435 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
datamodel-code-generator/datamodel-code-generator#5589 ·
Maintainers usually reply within 1 day
-
Shell: serve the home app's Cookie Policy on a hard reloadPossibly taken @2001anshkaushik claimed this today. Openfeature module:ai
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
rocketride-org/rocketride-server#2899 · 1 comment ·
Maintainers usually reply within 2 days
-
GEFSv12
Difficulty 2/5 1-3 hours Newbie friendliness 70/100