bug: A config reload stops the old pipeline while requests are still using it

オープン
  • 何か問題がありますか？
#1,624 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
go
領域
backend

調査の方向性

Start with core/reloader/reloader.go and the drain goroutine, then trace how Holder.Run pins a pipeline and how requests reach RunFinish; also find how WebSocket turns hold pipelines. The issue proposes stopping old pipelines when their holders finish, with a maximum wait and logging for remaining requests. Done means requests can finish on a live pipeline after reload, while the ceiling still ensures abandoned pipelines are eventually stopped.

索引モデルが issue の本文から書いたものです。

説明

bug

Problem

A config reload stops the old pipelines on a fixed timer: 30 seconds after the swap, whether or not anything is still using them (core/reloader/reloader.go, the drain goroutine after pipelines swapped). Stop calls each plugin's Shutdown.

Since #1561, a request keeps the pipeline it started on: Holder.Run pins it on the request context, and every later phase (RunResponse, RunResponseFrame, RunFinish, Resend) runs on that pin. That fixed the crash #1561 reported. But it means any request still open 30 seconds after a reload finishes on a pipeline whose plugins have shut down. Long requests are ordinary here:

  • a long SSE inference stream;
  • a CONNECT tunnel, whose Finishers run when it closes;
  • a Codex Responses WebSocket turn (#1467), which keeps the pipeline it started on.

#1561 handled the one plugin where this skipped policy: cpex now follows fail_open when it is not ready. It did not change when the reloader stops a pipeline.

What happens today

Among plugins that tear something down in Shutdown, these are the ones a request reaches after Stop, read from the code (not reproduced):

  • lineage (OnFinish): the request's response span is dropped, since the tracer provider is shut down and the OTel SDK ignores spans that end after it.
  • session-budget (last frame): the tokens reach the old instance's cache, and the Redis write fails with a WARN. The new instance never sees them.
  • script (frames, last frame): the worker is shared with the new build when the script did not change. A changed script's old worker has stopped, so the hook fails, and under fail_closed that is a late, recorded refusal.
  • cpex: denies or allows per fail_open (#1561).

Nothing crashes, and nothing the client receives changes. The laptop's default pipeline (inference-parser, mcp-parser, a2a-parser, tool-prune) has none of these.

Possible approach

Stop an old pipeline when nothing holds it, not after a fixed delay:

  • Count holders: increment when Holder.Run pins the pipeline, and decrement after the request's RunFinish. Paths that pin a pipeline outside Holder.Run (the WebSocket turns) would hold it the same way.
  • The reloader waits for the count to reach zero, with a ceiling (minutes, not seconds), so a tunnel left open for hours cannot keep a pipeline's resources forever. Past the ceiling it stops the pipeline as it does today, and logs how many requests were still on it.

Raised in

CodeRabbit's review of #1467, about WebSocket turns. The same exposure exists on main for every long-lived request, so it is tracked here rather than fixed in that PR.

主要言語
Go
スター
23
フォーク
44
平均マージ
16時間 37分
マージ済み PR（30日）
250

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

rossoctl/cortex のほかの issue

rossoctl/cortex の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。