🐛 Bug Report: OpenAI Responses: with_raw_response.create() raises when the body can't be parsed, though the uninstrumented call returns

Open Beginner friendly
#4,717 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Start in v1/responses_wrappers.py at the sync and async parse_response calls around lines 674 and 867, then inspect the wrapper's existing fallback when building TracedData fails. Add coverage using the issue's mock HTTP transport for both paths; done means malformed JSON returns the raw response without raising or recording metrics.

Written by the indexing model from the issue text.

Description

bug python
Which component is this bug for?

OpenAI Instrumentation

📜 Description

client.responses.with_raw_response.create() is meant to return the raw HTTP response without parsing the body. The user decides whether to call .parse().

The Responses wrapper in v1/responses_wrappers.py calls parse_response(response) itself, so it can read the response ID, model and usage. That call sits outside the wrapper's try/except (around line 674 for sync and line 867 for async). If the body can't be parsed, the exception from our .parse() goes up to the user. The same call returns normally when the instrumentation is off.

The async path (async_parse_response) has the same problem.

👟 Reproduction steps
import httpx, openai
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

def bad(req):
    return httpx.Response(200, headers={"content-type": "application/json"}, content=b"{not json")

def call():
    c = openai.OpenAI(api_key="k", max_retries=0,
                      http_client=httpx.Client(transport=httpx.MockTransport(bad)))
    return c.responses.with_raw_response.create(model="m", input="hi")

print(type(call()).__name__)   # LegacyAPIResponse
OpenAIInstrumentor().instrument()
call()                         # raises JSONDecodeError
👍 Expected behavior

The instrumentation never changes what the call returns or raises. If it can't parse the raw response, it should return the response untraced and record no metrics. The wrapper already does this when building TracedData fails (except Exception: return response).

👎 Actual Behavior with Screenshots
uninstrumented: returned LegacyAPIResponse status 200
instrumented:   raised JSONDecodeError
📃 Provide any additional context for the Bug.
  • This has been there since the first Responses instrumentation (#3052).
  • CodeRabbit raised it on #4713. It suggested recording duration and error.type and then re-raising. That would still break the user's call, and it would count a successful API call as a failed operation. So the fix is to guard the parse, not to record it as an error.
  • Suggested fix: wrap parse_response / async_parse_response in try/except Exception: return response in both the sync and async wrappers. Add a test using the mock transport above that checks the instrumented call returns the same object as the uninstrumented one.
  • Related: #4504, #4551, #4476 (other with_raw_response issues in the Responses wrapper).
👀 Have you spent some time to check if this bug has been raised before?
  • I checked and didn't find similar issue
Dominant language
Python
Stars
7.5k
Forks
1.2k
Avg merge
2d 6h
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from traceloop/openllmetry

All issues in traceloop/openllmetry

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.