🐛 Bug Report: OpenAI Responses: with_raw_response.create() raises when the body can't be parsed, though the uninstrumented call returns
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- backend-api-design, observability, testing
Research direction
Start in v1/responses_wrappers.py at the sync and async parse_response calls around lines 674 and 867, then inspect the wrapper's existing fallback when building TracedData fails. Add coverage using the issue's mock HTTP transport for both paths; done means malformed JSON returns the raw response without raising or recording metrics.
Written by the indexing model from the issue text.
Description
Which component is this bug for?
OpenAI Instrumentation
📜 Description
client.responses.with_raw_response.create() is meant to return the raw HTTP response without parsing the body. The user decides whether to call .parse().
The Responses wrapper in v1/responses_wrappers.py calls parse_response(response) itself, so it can read the response ID, model and usage. That call sits outside the wrapper's try/except (around line 674 for sync and line 867 for async). If the body can't be parsed, the exception from our .parse() goes up to the user. The same call returns normally when the instrumentation is off.
The async path (async_parse_response) has the same problem.
👟 Reproduction steps
import httpx, openai
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
def bad(req):
return httpx.Response(200, headers={"content-type": "application/json"}, content=b"{not json")
def call():
c = openai.OpenAI(api_key="k", max_retries=0,
http_client=httpx.Client(transport=httpx.MockTransport(bad)))
return c.responses.with_raw_response.create(model="m", input="hi")
print(type(call()).__name__) # LegacyAPIResponse
OpenAIInstrumentor().instrument()
call() # raises JSONDecodeError
👍 Expected behavior
The instrumentation never changes what the call returns or raises. If it can't parse the raw response, it should return the response untraced and record no metrics. The wrapper already does this when building TracedData fails (except Exception: return response).
👎 Actual Behavior with Screenshots
uninstrumented: returned LegacyAPIResponse status 200
instrumented: raised JSONDecodeError
📃 Provide any additional context for the Bug.
- This has been there since the first Responses instrumentation (#3052).
- CodeRabbit raised it on #4713. It suggested recording duration and
error.typeand then re-raising. That would still break the user's call, and it would count a successful API call as a failed operation. So the fix is to guard the parse, not to record it as an error. - Suggested fix: wrap
parse_response/async_parse_responseintry/except Exception: return responsein both the sync and async wrappers. Add a test using the mock transport above that checks the instrumented call returns the same object as the uninstrumented one. - Related: #4504, #4551, #4476 (other
with_raw_responseissues in the Responses wrapper).
👀 Have you spent some time to check if this bug has been raised before?
- I checked and didn't find similar issue
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 1.2k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 2
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from traceloop/openllmetry
-
🐛 Bug Report: Incomplete cache token migration after #4243 breaks Bedrock and Groq instrumentationPossibly taken @Quantum-Enigma-dot claimed this 21 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
traceloop/openllmetry#4665 · 1 comment ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: OpenAI instrumentation drops gen_ai.usage.reasoning_tokens on streamed chat completionsPossibly taken @jayfeng7 claimed this 23 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
traceloop/openllmetry#4660 · 1 comment ·
Maintainers usually reply within 1 day
-
Bug Report: Failed dataset deletes still remove local rows and columnsPossibly taken @inchang-ing claimed this 32 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
traceloop/openllmetry#4630 ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: LiteLLM instrumentation does not record seed, stop, frequency_penalty, presence_penaltyPossibly taken @shantam77 claimed this 38 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
traceloop/openllmetry#4598 · 2 comments ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: LiteLLM instrumentation never ends the span when a call is interrupted (KeyboardInterrupt, asyncio cancellation)Possibly taken @rupak-eng claimed this 39 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
traceloop/openllmetry#4597 · 1 comment ·
Maintainers usually reply within 1 day
All issues in traceloop/openllmetry
Similar issues
-
Internal ErrorOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
frappe/frappe_docker#2071 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
The-Pocket/PocketFlow#161 ·
-
comp/desktop P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#157702 ·
Maintainers usually reply within 1 day
-
scibench_physics / scibench_chemistry: a correct answer scores 0 when the unit has a power of tenPossibly taken @BahaaMattar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
EvolvingLMMs-Lab/lmms-eval#1699 ·
Maintainers usually reply within 1 day
-
project_inputs.html uses undefined file variable for dropdown IDs, lacks visual tag badges, and has empty delete labelsPossibly taken @Rucha0901 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aboutcode-org/scancode.io#2354 ·
Maintainers usually reply within 1 day