bug: A config reload stops the old pipeline while requests are still using it

Đã đóng
  • Có gì không ổn?
#1,624 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@evaline-ju đang làm issue này rồi.

Từ ngày 4/11/2026.

  • #1661 của @evaline-ju — đang mở

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
45/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go
Lĩnh vực
backend

Hướng nghiên cứu

Start with core/reloader/reloader.go and the drain goroutine, then trace how Holder.Run pins a pipeline and how requests reach RunFinish; also find how WebSocket turns hold pipelines. The issue proposes stopping old pipelines when their holders finish, with a maximum wait and logging for remaining requests. Done means requests can finish on a live pipeline after reload, while the ceiling still ensures abandoned pipelines are eventually stopped.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug

Problem

A config reload stops the old pipelines on a fixed timer: 30 seconds after the swap, whether or not anything is still using them (core/reloader/reloader.go, the drain goroutine after pipelines swapped). Stop calls each plugin's Shutdown.

Since #1561, a request keeps the pipeline it started on: Holder.Run pins it on the request context, and every later phase (RunResponse, RunResponseFrame, RunFinish, Resend) runs on that pin. That fixed the crash #1561 reported. But it means any request still open 30 seconds after a reload finishes on a pipeline whose plugins have shut down. Long requests are ordinary here:

  • a long SSE inference stream;
  • a CONNECT tunnel, whose Finishers run when it closes;
  • a Codex Responses WebSocket turn (#1467), which keeps the pipeline it started on.

#1561 handled the one plugin where this skipped policy: cpex now follows fail_open when it is not ready. It did not change when the reloader stops a pipeline.

What happens today

Among plugins that tear something down in Shutdown, these are the ones a request reaches after Stop, read from the code (not reproduced):

  • lineage (OnFinish): the request's response span is dropped, since the tracer provider is shut down and the OTel SDK ignores spans that end after it.
  • session-budget (last frame): the tokens reach the old instance's cache, and the Redis write fails with a WARN. The new instance never sees them.
  • script (frames, last frame): the worker is shared with the new build when the script did not change. A changed script's old worker has stopped, so the hook fails, and under fail_closed that is a late, recorded refusal.
  • cpex: denies or allows per fail_open (#1561).

Nothing crashes, and nothing the client receives changes. The laptop's default pipeline (inference-parser, mcp-parser, a2a-parser, tool-prune) has none of these.

Possible approach

Stop an old pipeline when nothing holds it, not after a fixed delay:

  • Count holders: increment when Holder.Run pins the pipeline, and decrement after the request's RunFinish. Paths that pin a pipeline outside Holder.Run (the WebSocket turns) would hold it the same way.
  • The reloader waits for the count to reach zero, with a ceiling (minutes, not seconds), so a tunnel left open for hours cannot keep a pipeline's resources forever. Past the ceiling it stops the pipeline as it does today, and logs how many requests were still on it.

Raised in

CodeRabbit's review of #1467, about WebSocket turns. The same exposure exists on main for every long-lived request, so it is tracked here rather than fixed in that PR.

Ngôn ngữ chính
Go
Star
23
Fork
44
Merge trung bình
16 giờ 8 phút
Pull request đã merge (30 ngày)
234

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của rossoctl/cortex

Tất cả issue của rossoctl/cortex

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.