[Engine] serve: a stream the engine could not finish sends no finish_reason, so the cause the engine named is lost

オープン 初心者向け
  • 何か問題がありますか？
#5,633 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 5 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
76/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cpp, python
領域
backend, backend-api-design

調査の方向性

serve/server.py の make_handler にあるストリーミング例外ハンドラーから始めてください。報告では4つのエラーパスが示され、終了チャンクの案も含まれています。既存のストリーミングテスト serve/test_fatal_recovery.py を読み、エラーレスポンスとストリームの終了をどのように検証しているか確認してください。報告されたストリーム途中の各失敗に finish_reason を含む終了チャンクが含まれ、既存のエラーオブジェクトと [DONE] も引き続き存在すれば完了です。

索引モデルが issue の本文から書いたものです。

説明

Strata app 0.2.12 (146df3c4) with the patch below on top · engine 0.1.58 (cuda13) · PC RTX 5090 / 64 GB RAM / Windows 11 (Korean locale)

What happens

When the engine fails after the SSE headers have already gone out, serve/server.py sends the error object and [DONE], and nothing that closes the stream:

data: {"error": {"type": "server_error", "message": "the engine stopped unexpectedly (exit code 1) ..."}}

data: [DONE]

A client written for the OpenAI protocol reads chunks until one carries a finish_reason. It never gets one, so the best it can say is:

stream ended without finish_reason

The cause the engine did name is sitting in the error object above, but a chunk-protocol client does not look there. On my PC that turned a named failure (the watchdog of #29, RAM) into a message that reads exactly like a dead connection - and the only way to make my client show anything at all was requireFinishReason: false, which hides the error instead of reporting it.

Four paths do this, all in make_handler's streaming block:

exception error object it sends
EngineDied server_error
QueueWaitExpired rate_limit_error / queue_wait_timeout
StructuredOutputError structured_output_failed
ValueError (the engine's ERR after the stream started) server_error

What I expected

The error object, then a chunk that closes the stream, so a client can report the cause instead of the absence of one:

data: {"error": {...}}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"error"}]}

data: [DONE]

finish_reason: "error" is outside OpenAI's enum. If you would rather stay inside it, "stop" plus the error object fixes the same client problem - what matters is that the stream closes and the engine's reason survives. I picked "error" because a stream that closes as if it had finished is the one failure mode a client cannot tell from success.

The patch I am running

serve/server.py 
@@ -7156,6 +7156,15 @@ def prompt_progress(svc: Service):
             "time_ms": int(getattr(engine, "progress_ms", 0) or 0)}
 
 
+def openai_error_chunk(svc: Service, req: dict) -> dict:
+    """The chunk that closes a stream the engine could not finish.  A stream that ends with only an `{"error": ..}`
+    object and [DONE] carries no finish_reason, so a client written for the OpenAI protocol cannot tell it apart
+    from a stream that was cut short: it reports "stream ended without finish_reason" and the cause the engine gave
+    is lost.  This is the chunk that says "the answer stopped because the server failed"."""
+    return {"id": "chatcmpl-" + uuid.uuid4().hex[:24], "object": "chat.completion.chunk", "created": int(time.time()),
+            "model": svc.model_for(req), "choices": [{"index": 0, "delta": {}, "finish_reason": "error"}]}
+
+
 def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel, run=None, force=None):
@@ -8575,6 +8584,15 @@ def make_handler(svc: Service):
                 except OSError:
                     pass
 
+        def _sse_error(self, err: dict, req: dict) -> None:
+            """An error after the SSE headers went out (so no 400 is possible): the error object, then a chunk that
+            closes the stream.  The web app reads the error object; an OpenAI client reads the finish_reason and
+            reports the failure as one - without it all it can say is that the stream ended for no reason."""
+            self._note(error=err["error"])
+            self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\n" +
+                             b"data: " + json.dumps(openai_error_chunk(svc, req)).encode() + b"\n\n" +
+                             b"data: [DONE]\n\n")
+
         def _capture(self, items, api):
@@ -8689,21 +8707,15 @@ def make_handler(svc: Service):
                 svc.note_client_disconnect(cancel)           # client went away: stop the engine
                 chunks.close()
             except EngineDied as e:                          # mid-stream: say so, then end the stream properly
-                err = {"error": {"type": "server_error", "message": died_note(e)}}
-                self._note(error=err["error"])
-                self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\ndata: [DONE]\n\n")
-            except QueueWaitExpired as e:
-                err = {"error": {"type": "rate_limit_error", "code": "queue_wait_timeout", "message": str(e)}}
-                self._note(error=err["error"])
-                self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\ndata: [DONE]\n\n")
+                self._sse_error({"error": {"type": "server_error", "message": died_note(e)}}, req)
+            except QueueWaitExpired as e:                    # it waited for a free slot too long: the answer
+                self._sse_error({"error": {"type": "rate_limit_error", "code": "queue_wait_timeout",   # never started
+                                           "message": str(e)}}, req)
             except StructuredOutputError as e:
-                err = {"error": {"type": "structured_output_failed", "code": "structured_output_failed", "message": str(e)}}
-                self._note(error=err["error"])
-                self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\ndata: [DONE]\n\n")
+                self._sse_error({"error": {"type": "structured_output_failed", "code": "structured_output_failed",
+                                           "message": str(e)}}, req)
             except ValueError as e:                          # the engine's ERR after the stream started: the
-                err = {"error": {"type": "server_error", "message": str(e)}}   # headers are sent, so no 400 now
-                self._note(error=err["error"])
-                self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\ndata: [DONE]\n\n")
+                self._sse_error({"error": {"type": "server_error", "message": str(e)}}, req)  # no 400 is possible now

The error object is still the first thing on the wire, so the web app keeps reading what it reads today; only the closing chunk is new.

Test

One assertion added to the test that already covers this stream, serve/test_fatal_recovery.py:

             text=post()
             self.assertIn('"error"',text);self.assertTrue(text.rstrip().endswith('data: [DONE]'))
+            self.assertIn('"finish_reason": "error"',text)   # an OpenAI client needs it to report the cause, not
+                                                             # "the stream ended without finish_reason"
             with urllib.request.urlopen(base+'/health',timeout=5) as r:self.assertFalse(json.load(r)['loaded'])

Results on this PC:

  • python -X utf8 -m unittest serve.test_fatal_recovery from a copy on an ASCII path: Ran 8 tests in 5.771s → OK with the patch and the extra assertion.
  • the same command from the repo's own Korean path: FAILED (errors=8) - that is the batch-file encoding of #5632, not this change. The same setUp lines, run outside the repo in UTF-8 mode, give the same EngineDied: the engine exited before it was ready (exit code 1); writing that batch file with encoding='mbcs' makes it pass.
  • the whole serve suite on 0.2.12 with this patch: Ran 1510 tests in 710.156s → FAILED (errors=1), and that one error is #5631.
主要言語
C++
スター
11.6k
フォーク
1k
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Niko1221/Strata のほかの issue

Niko1221/Strata の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。