🐛 Bug Report: Traces go missing when a model reports 0 tokens
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- observability
Research direction
Start by running the reproduction script in the issue, which uses a mocked OpenAI transport returning zero token usage, and confirm both span lists are empty. The two instrumentors involved are LangchainInstrumentor and OpenAIInstrumentor, so compare how each handles a zero token count in the usage data. Done when both spans appear with 0 tokens and later calls keep being traced.
Written by the indexing model from the issue text.
Description
Which component is this bug for?
Langchain Instrumentation
📜 Description
When a model returns 0 in its token usage, the Langchain span is never sent, and later LLM calls stop being traced too.
Local models and proxies often report 0 tokens, so this is easy to hit.
This is what i assume and reviewed, correct me if i am wrong
👟 Reproduction steps
No API key needed, it uses a fake OpenAI server that returns 0 tokens:
import httpx
import openai
from langchain_openai import ChatOpenAI
from opentelemetry.instrumentation.langchain import LangchainInstrumentor
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
LangchainInstrumentor().instrument(tracer_provider=provider)
OpenAIInstrumentor().instrument(tracer_provider=provider)
# fake OpenAI server that reports 0 tokens
http = httpx.Client(transport=httpx.MockTransport(lambda request: httpx.Response(200, json={
"id": "1", "object": "chat.completion", "created": 0, "model": "m",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "hi"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
})))
ChatOpenAI(model="m", api_key="x", base_url="http://mock/v1", http_client=http).invoke("hi")
print("langchain spans:", [s.name for s in exporter.get_finished_spans()])
exporter.clear()
openai.OpenAI(api_key="x", base_url="http://mock/v1", http_client=http).chat.completions.create(
model="m", messages=[{"role": "user", "content": "hi"}]
)
print("openai spans:", [s.name for s in exporter.get_finished_spans()])
👍 Expected behavior
langchain spans: ['ChatOpenAI.chat']
openai spans: ['openai.chat']
👎 Actual Behavior with Screenshots
langchain spans: []
openai spans: []
If you change the usage to non-zero numbers, both spans show up as expected.
🤖 Python Version
3.11
📃 Provide any additional context for the Bug.
No response
👀 Have you spent some time to check if this bug has been raised before?
- I checked and didn't find similar issue
Are you willing to submit PR?
Yes I am willing to submit a PR!
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 1.2k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 2
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from traceloop/openllmetry
-
bug python
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
traceloop/openllmetry#4717 ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: Incomplete cache token migration after #4243 breaks Bedrock and Groq instrumentationPossibly taken @Quantum-Enigma-dot claimed this 21 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
traceloop/openllmetry#4665 · 1 comment ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: OpenAI instrumentation drops gen_ai.usage.reasoning_tokens on streamed chat completionsPossibly taken @jayfeng7 claimed this 23 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
traceloop/openllmetry#4660 · 1 comment ·
Maintainers usually reply within 1 day
-
Bug Report: Failed dataset deletes still remove local rows and columnsPossibly taken @inchang-ing claimed this 32 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
traceloop/openllmetry#4630 ·
Maintainers usually reply within 1 day
-
🐛 Bug Report: LiteLLM instrumentation does not record seed, stop, frequency_penalty, presence_penaltyPossibly taken @shantam77 claimed this 38 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
traceloop/openllmetry#4598 · 2 comments ·
Maintainers usually reply within 1 day
All issues in traceloop/openllmetry
Similar issues
-
opensubtitlescom: provider rejects pt-PT even though its own converter maps itPossibly taken @paulo-0308 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Diaoul/subliminal#1405 ·
-
Internal ErrorOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
frappe/frappe_docker#2071 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
The-Pocket/PocketFlow#161 ·
-
comp/desktop P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#157702 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
zai-org/zcode-plugins#202 ·
Maintainers usually reply within 1 day