bug: A config reload stops the old pipeline while requests are still using it

已关闭
  • 哪里不对？
#1,624 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@evaline-ju 已经在做这个了。

开始于 2026年11月4日。

  • #1661 来自 @evaline-ju —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
45/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
go
领域
backend

调研方向

Start with core/reloader/reloader.go and the drain goroutine, then trace how Holder.Run pins a pipeline and how requests reach RunFinish; also find how WebSocket turns hold pipelines. The issue proposes stopping old pipelines when their holders finish, with a maximum wait and logging for remaining requests. Done means requests can finish on a live pipeline after reload, while the ceiling still ensures abandoned pipelines are eventually stopped.

由索引模型根据 Issue 内容生成。

描述

bug

Problem

A config reload stops the old pipelines on a fixed timer: 30 seconds after the swap, whether or not anything is still using them (core/reloader/reloader.go, the drain goroutine after pipelines swapped). Stop calls each plugin's Shutdown.

Since #1561, a request keeps the pipeline it started on: Holder.Run pins it on the request context, and every later phase (RunResponse, RunResponseFrame, RunFinish, Resend) runs on that pin. That fixed the crash #1561 reported. But it means any request still open 30 seconds after a reload finishes on a pipeline whose plugins have shut down. Long requests are ordinary here:

  • a long SSE inference stream;
  • a CONNECT tunnel, whose Finishers run when it closes;
  • a Codex Responses WebSocket turn (#1467), which keeps the pipeline it started on.

#1561 handled the one plugin where this skipped policy: cpex now follows fail_open when it is not ready. It did not change when the reloader stops a pipeline.

What happens today

Among plugins that tear something down in Shutdown, these are the ones a request reaches after Stop, read from the code (not reproduced):

  • lineage (OnFinish): the request's response span is dropped, since the tracer provider is shut down and the OTel SDK ignores spans that end after it.
  • session-budget (last frame): the tokens reach the old instance's cache, and the Redis write fails with a WARN. The new instance never sees them.
  • script (frames, last frame): the worker is shared with the new build when the script did not change. A changed script's old worker has stopped, so the hook fails, and under fail_closed that is a late, recorded refusal.
  • cpex: denies or allows per fail_open (#1561).

Nothing crashes, and nothing the client receives changes. The laptop's default pipeline (inference-parser, mcp-parser, a2a-parser, tool-prune) has none of these.

Possible approach

Stop an old pipeline when nothing holds it, not after a fixed delay:

  • Count holders: increment when Holder.Run pins the pipeline, and decrement after the request's RunFinish. Paths that pin a pipeline outside Holder.Run (the WebSocket turns) would hold it the same way.
  • The reloader waits for the count to reach zero, with a ceiling (minutes, not seconds), so a tunnel left open for hours cannot keep a pipeline's resources forever. Past the ceiling it stops the pipeline as it does today, and logs how many requests were still on it.

Raised in

CodeRabbit's review of #1467, about WebSocket turns. The same exposure exists on main for every long-lived request, so it is tracked here rather than fixed in that PR.

主要语言
Go
星标
23
派生
44
平均合并
14 小时 38 分钟
30 天内合并 PR
254

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

rossoctl/cortex 的其他 Issue

查看 rossoctl/cortex 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。