Cloud first-use: GLiNER2.5-Decide waits 21s before PROVISIONING; clarify readiness and latency expectations

オープン
#851 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
25/100
issue の種類
ドキュメント
明瞭さ
おおむね明確
活発さ
活発

調査の方向性

Start with the model page that shows the 13.1 ms figure and the Cloud retry docs at sie-relay#retry-while-capacity-starts and the /v1/models state section of the API reference. Reproduce the 503 PROVISIONING response from POST /v1/decisions using the bodies in the issue, and separate time-to-headers from time-to-full-body. Done means the docs state the measurement boundary and first-use expectations, and the 21-second pre-refusal wait is either explained or filed as a separate server-side investigation.

索引モデルが issue の本文から書いたものです。

説明

Small Cloud requests had substantially different first-use and subsequent latency. In particular, fastino/GLiNER2.5-Decide took 21.306 s before returning 503 PROVISIONING with Retry-After: 60. Later identical requests succeeded in 233–244 ms, with 7.6–10.6 ms reported inference time.

Provisioning is documented and expected. This report concerns the first-use experience: a client spends 21 seconds waiting before learning it must wait again, and a catalog entry does not make the likely wait clear. The impact is an unpredictable delay before an interactive search/classification feature can produce its first result.

Observed against https://api.superlinked.com on 2026-10-25 UTC, from one remote desktop client. Successful initial responses reported SIE version 0.10.1.

Model / tiny input Initial observed call Two subsequent calls, response fully read Subsequent X-Inference-Time
Qwen/Qwen3-Embedding-4B, 2 short texts 200, 25.192 s 563–783 ms 25.5–26.6 ms
Qwen/Qwen3-Reranker-4B, 2 pairs 200, 3.971 s 295–372 ms 28.6–28.8 ms
fastino/GLiNER2.5-Decide, 1 choice question 503, 21.306 s; retry after 60 s 200, 233–244 ms 7.6–10.6 ms
zai-org/GLM-OCR, small printed image 200, 4.000 s 287–392 ms 52.2–59.9 ms
docling, 1-page PDF 200, 11.366 s 391–419 ms 141.5–156.7 ms

The text calls started around 16:54:11; OCR/PDF around 16:55:34. The first successful decision call, at 16:59:20, took 1.441 s and reported 813.5 ms inference. The two subsequent rounds ran at 17:02:39–17:02:44, one request at a time, with no automatic retries; all 12 requests across six models returned 200. The sixth model, nvidia/parakeet-tdt-0.6b-v3, processed a valid 3.446-second clip in 370–508 ms subsequently (92.1–100 ms inference). Its earlier valid call is excluded from the first-use comparison because a previous invalid fixture had already contacted that model.

These are observations, not percentiles or a controlled cold/warm benchmark. Initial text calls ran concurrently across three different models; subsequent calls were sequential. Capacity was not forcibly evicted, other service load was not controlled, and the manual gap before the successful decision call does not measure provisioning duration. Timing headers are reported separately, not added together.

The GLiNER2.5-Decide page displays 13.1 ms per message, and qualifies the test farther down as a GPU server with the client alongside. The subsequent inference times above are compatible with that scale. The missing expectation is Cloud request latency and first-use waiting, rather than evidence that steady-state GPU execution is slow.

Before these calls, /v1/models returned state: "available", loaded: false for both embedding and decision models. That is consistent with catalog availability, but the API reference recommends using state without explaining this distinction in the model-list section.

Two concrete improvements would help:

  1. Put the measurement boundary beside the model page's headline time: colocated benchmark versus Cloud request latency, plus model-specific first-use/capacity-start expectations where known.
  2. Make first-use readiness actionable: clearly distinguish catalogued/loaded/starting states and investigate whether the 21-second wait before a retryable refusal can be shortened. Clarify whether a returned 60-second hint is a default, an estimate, or the compatibility-route cap, so a caller can present a useful waiting state.

For reproduction, send these exact bodies with normal Cloud authentication, Content-Type: application/json and Accept: application/json. Authentication is intentionally omitted. Measure time to headers and to the fully read response separately. If capacity is starting, honor Retry-After before one manual follow-up; do not force eviction or repeatedly poll inference.

POST /v1/encode/Qwen%2FQwen3-Embedding-4B (220 bytes):

{"items":[{"id":"garden","text":"Plant basil on Saturday and water the seedlings every morning."},{"id":"software","text":"The development server crashes when I edit a source file."}],"params":{"output_types":["dense"]}}

POST /v1/decisions (248 bytes):

{"model":"fastino/GLiNER2.5-Decide","input":"The development server crashes when I edit a source file.","questions":[{"type":"choice","name":"topic","instructions":"What is the topic?","choices":[{"value":"gardening"},{"value":"software issue"}]}]}

Request correlation:

  • Initial embedding, 16:54:11.347: 01a173fb-e9e9-7f00-907b-4a443109c2d2.
  • Initial decision refusal, 16:54:11.366: PROVISIONING, Retry-After: 60, served locally; no request ID in the captured response headers.
  • First successful decision, 16:59:20.836: 01a17400-a2f6-74c0-87c9-0e40ed0d5c47.
  • Subsequent decision, 17:02:43.375: 01a17403-b912-7c62-a02e-b4541053b455.

Related: #293 concerned workers repeatedly scaling down during loading; these models subsequently served successfully. #688/#698 already improved provisioning messages and retry hints. This report asks about the remaining Cloud first-use experience, not reopening those failures.

主要言語
Python
スター
3.5k
フォーク
328
平均マージ
16時間 25分
マージ済み PR(30日)
121

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

superlinked/sie のほかの issue

superlinked/sie の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。