event_loop_blocked can emit a parentless span when the stall is sampled in job_task after session close

Đang mở
  • Có gì không ổn?
#7,646 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@jipeng6036-del đang làm issue này rồi.

Từ ngày 6/10/2026.

  • #7649 của @jipeng6036-del — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
observability

Hướng nghiên cứu

Start with telemetry/loop_monitor.py, especially _emit_span around line 446, then trace the task and session context setup in ipc/job_proc_lazy_main.py and voice/agent_session.py at the cited locations. Reproduce or test the outer job_task stall after session close and inspect why the report-context fallback does not provide a parent. Done means resolving the cause so the emitted span has the intended parent, or is skipped when none is valid.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

An event_loop_blocked span can be emitted with no parent, becoming a stray root, when the stall is sampled in the outer job_task after the agent session has closed. The documented third-tier fallback in _emit_span — "else the report context (job_entrypoint)" — does not appear to be reached in this path.

Introduced by #7128 (1ac4183fc), so this is main-only; loop_monitor.py is not present in 1.8.4 or earlier.

Observed

In one 579 s voice session on main (8a677a62), 5 event_loop_blocked spans were emitted. 4 parented correctly. The 5th did not:

sampled in task parent
job_user_entrypoint job_entrypoint ✅
LLM._main_task the task's own span ✅
(task span) the task's own span ✅
(task span) the task's own span ✅
job_task none — stray root ❌

The orphan fired at t+561.18 s, in the same millisecond agent_session and session_close ended (t+561.17 s), and ~18 s before job_entrypoint itself ended (t+579.13 s). So the root span was still open and still a valid parent at the time.

Why the fallback chain doesn't cover it

_emit_span (telemetry/loop_monitor.py:446) selects a parent in three tiers, per its own comment — "under the span the blocked task was in, else the session root, else the report context (job_entrypoint)":

if report.parent_span_context is not None:
    parent = trace.set_span_in_context(trace.NonRecordingSpan(report.parent_span_context))
else:
    parent = session_context.session_root_context()
span = tracer.start_span(SPAN_NAME, context=parent, start_time=start_ns, attributes=attributes)

Tier 1 is unavailable for job_task. _run_job_task is created at ipc/job_proc_lazy_main.py:323, and only the inner job_user_entrypoint task (:359) has its body wrapped in tracer.use_span(job_span, ...). The outer task is what creates job_span, never runs inside it, so a stall sampled there has no task span to inherit.

Tier 2 is cleared at session close. voice/agent_session.py:1356 sets self._root_span_context = None, so session_root_context() returns None from that point on.

Tier 3 then appears not to fire. With parent = None, start_span(context=None) should resolve to the ambient context, and _emit runs inside self._report_context.run(_emit) whose snapshot was taken inside use_span(job_span) — so job_entrypoint should be current. The emitted span nonetheless has no parent. I have not pinned down why, and did not want to guess at a patch.

Note that event_loop_blocked is the only span in the system whose parentage is reconstructed rather than inherited — it originates on the watchdog thread, which has no ambient OTel context of its own. That is what set_report_context(contextvars.copy_context()) at job_proc_lazy_main.py:347 exists for, and it is why this path has no natural backstop.

Impact

Low. At most one diagnostic span per session, and it still carries the same trace id, so nothing else is detached. But a backend that groups by trace membership renders it as a separate root, which is noise in exactly the view #7533 set out to clean up.

Possible directions

Offered as suggestions rather than a diagnosis, since I could not confirm the tier-3 failure:

  1. In _emit_span, fall through to the ambient/report context when session_root_context() yields no span with a valid context, instead of passing its result as context= unconditionally — this makes the documented tier 3 actually reachable.
  2. Wrap _run_job_task's body in use_span(job_span) so tier 1 also covers stalls sampled in the outer job task.
  3. Alternatively, skip emitting when no valid parent can be resolved — _emit_span already returns early when there is no job context (loop_monitor.py:442), on the same reasoning that "a root span here would be a stray trace".
Environment
  • livekit-agents main @ 8a677a62, Python 3.13
  • OTLP/HTTP export, voice (cascade) pipeline, single session per job process
Ngôn ngữ chính
Python
Star
14.6k
Fork
3.9k
Merge trung bình
2 ngày 23 giờ
Pull request đã merge (30 ngày)
263

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của livekit/agents

Tất cả issue của livekit/agents

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.