Annotation.create() cannot tell a caller the write didn't land
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start with packages/traceloop-sdk/traceloop/sdk/client/http.py and packages/traceloop-sdk/traceloop/sdk/annotation/base_annotation.py, then trace the sibling get, delete, and put paths plus the datasets/* and annotation/user_feedback.py callers. Decide the intended delivered, refused, and unreachable outcomes before changing the client contract; done means callers can distinguish write results without relying on stdout or an indistinguishable None.
Written by the indexing model from the issue text.
Description
Title: Annotation.create() cannot tell a caller the write didn't land
Reading the SDK to understand how annotation writes report failure, and I think three
distinguishable outcomes are collapsing into one. Flagging it because the asymmetry looks
unintentional rather than by design — happy to be told it's deliberate.
packages/traceloop-sdk/traceloop/sdk/client/http.py — post() catches
requests.exceptions.RequestException, prints the formatted error, and returns None.
Because raise_for_status() runs inside that try, an HTTP 500 becomes a RequestException
too. So three different worlds produce the identical return:
- the annotation was written
- the network never reached you
- your server received it and explicitly rejected it
packages/traceloop-sdk/traceloop/sdk/annotation/base_annotation.py — create() calls
post(...) and discards the result. Its signature is -> None, so a caller has nothing to
inspect even if they wanted to.
What makes this look like an oversight rather than a policy: the same function is perfectly
willing to raise. It raises ValueError three times, for a missing annotation_task, a missing
entity_id, and empty tags. It will stop you for a malformed argument and stay silent when the
write itself is lost.
The error channel is also a print() to stdout rather than a logger or a warning. In a worker,
a scheduled job, or a container, that is nobody's alerting — human feedback data disappears behind
a red line no one reads.
Same shape on the sibling verbs, so it isn't one method: get → None, delete → False, and
put → None on failure but {} on an empty success, so even the boolean one can't separate
refused from unreachable.
And the part that argues against a blanket reading: evaluator.py is clean. It uses httpx
directly and raises on a non-200. So this isn't "the SDK swallows errors" — it's two client paths
in one SDK with opposite failure policies, and a caller's guarantees depend on which one they
happened to hit.
A fix that keeps the property you probably want: if the swallow is deliberate — and it very
plausibly is, a telemetry SDK that crashes the host app is worse than one that loses a write — then
the honest version is a return value that distinguishes delivered / refused / could-not-reach,
with create() handing it back rather than returning None. An SDK that won't take your app down
is right. One that can't tell you it lost your data is a different thing.
What I have not done, and it matters for how much weight to give this: I read this on main
through the API. I have not run it and have not reproduced a dropped write. Fifteen files
use this client — datasets/* and annotation/user_feedback.py among them, all write paths — and
I traced exactly one of them. The mechanism is certain for the annotation path; the blast radius is
inference on my part.
I do this class of work — checks that can't come out against you, guards that never fire, verdicts
with nothing behind them — at unreached.dev, and I'll say so plainly rather than pretend I'm only
here to be helpful. The issue stands either way and I'm happy to leave it at that.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 1.2k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 2
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from traceloop/openllmetry
-
🐛 Bug Report: OpenAI Responses: with_raw_response.create() raises when the body can't be parsed, though the uninstrumented call returnsPossibly taken A pull request linked to this issue is open or already merged. Openbug python
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
traceloop/openllmetry#4717 ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: Incomplete cache token migration after #4243 breaks Bedrock and Groq instrumentationPossibly taken @Quantum-Enigma-dot claimed this 21 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
traceloop/openllmetry#4665 · 1 comment ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: OpenAI instrumentation drops gen_ai.usage.reasoning_tokens on streamed chat completionsPossibly taken @jayfeng7 claimed this 23 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
traceloop/openllmetry#4660 · 1 comment ·
Maintainers usually reply within 1 day
-
Bug Report: Failed dataset deletes still remove local rows and columnsPossibly taken @inchang-ing claimed this 32 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
traceloop/openllmetry#4630 ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: LiteLLM instrumentation does not record seed, stop, frequency_penalty, presence_penaltyPossibly taken @shantam77 claimed this 38 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
traceloop/openllmetry#4598 · 2 comments ·
Maintainers usually reply within 1 day
All issues in traceloop/openllmetry
Similar issues
-
opensubtitlescom: provider rejects pt-PT even though its own converter maps itPossibly taken @paulo-0308 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Diaoul/subliminal#1405 ·
-
Internal ErrorOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
frappe/frappe_docker#2071 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
The-Pocket/PocketFlow#161 ·
-
comp/desktop P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#157702 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
zai-org/zcode-plugins#202 ·
Maintainers usually reply within 1 day