Cloud first-use: GLiNER2.5-Decide waits 21s before PROVISIONING; clarify readiness and latency expectations

Aperta
#851 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva

Direzione di ricerca

Start with the model page that shows the 13.1 ms figure and the Cloud retry docs at sie-relay#retry-while-capacity-starts and the /v1/models state section of the API reference. Reproduce the 503 PROVISIONING response from POST /v1/decisions using the bodies in the issue, and separate time-to-headers from time-to-full-body. Done means the docs state the measurement boundary and first-use expectations, and the 21-second pre-refusal wait is either explained or filed as a separate server-side investigation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Small Cloud requests had substantially different first-use and subsequent latency. In particular, fastino/GLiNER2.5-Decide took 21.306 s before returning 503 PROVISIONING with Retry-After: 60. Later identical requests succeeded in 233–244 ms, with 7.6–10.6 ms reported inference time.

Provisioning is documented and expected. This report concerns the first-use experience: a client spends 21 seconds waiting before learning it must wait again, and a catalog entry does not make the likely wait clear. The impact is an unpredictable delay before an interactive search/classification feature can produce its first result.

Observed against https://api.superlinked.com on 2026-10-25 UTC, from one remote desktop client. Successful initial responses reported SIE version 0.10.1.

Model / tiny input Initial observed call Two subsequent calls, response fully read Subsequent X-Inference-Time
Qwen/Qwen3-Embedding-4B, 2 short texts 200, 25.192 s 563–783 ms 25.5–26.6 ms
Qwen/Qwen3-Reranker-4B, 2 pairs 200, 3.971 s 295–372 ms 28.6–28.8 ms
fastino/GLiNER2.5-Decide, 1 choice question 503, 21.306 s; retry after 60 s 200, 233–244 ms 7.6–10.6 ms
zai-org/GLM-OCR, small printed image 200, 4.000 s 287–392 ms 52.2–59.9 ms
docling, 1-page PDF 200, 11.366 s 391–419 ms 141.5–156.7 ms

The text calls started around 16:54:11; OCR/PDF around 16:55:34. The first successful decision call, at 16:59:20, took 1.441 s and reported 813.5 ms inference. The two subsequent rounds ran at 17:02:39–17:02:44, one request at a time, with no automatic retries; all 12 requests across six models returned 200. The sixth model, nvidia/parakeet-tdt-0.6b-v3, processed a valid 3.446-second clip in 370–508 ms subsequently (92.1–100 ms inference). Its earlier valid call is excluded from the first-use comparison because a previous invalid fixture had already contacted that model.

These are observations, not percentiles or a controlled cold/warm benchmark. Initial text calls ran concurrently across three different models; subsequent calls were sequential. Capacity was not forcibly evicted, other service load was not controlled, and the manual gap before the successful decision call does not measure provisioning duration. Timing headers are reported separately, not added together.

The GLiNER2.5-Decide page displays 13.1 ms per message, and qualifies the test farther down as a GPU server with the client alongside. The subsequent inference times above are compatible with that scale. The missing expectation is Cloud request latency and first-use waiting, rather than evidence that steady-state GPU execution is slow.

Before these calls, /v1/models returned state: "available", loaded: false for both embedding and decision models. That is consistent with catalog availability, but the API reference recommends using state without explaining this distinction in the model-list section.

Two concrete improvements would help:

  1. Put the measurement boundary beside the model page's headline time: colocated benchmark versus Cloud request latency, plus model-specific first-use/capacity-start expectations where known.
  2. Make first-use readiness actionable: clearly distinguish catalogued/loaded/starting states and investigate whether the 21-second wait before a retryable refusal can be shortened. Clarify whether a returned 60-second hint is a default, an estimate, or the compatibility-route cap, so a caller can present a useful waiting state.

For reproduction, send these exact bodies with normal Cloud authentication, Content-Type: application/json and Accept: application/json. Authentication is intentionally omitted. Measure time to headers and to the fully read response separately. If capacity is starting, honor Retry-After before one manual follow-up; do not force eviction or repeatedly poll inference.

POST /v1/encode/Qwen%2FQwen3-Embedding-4B (220 bytes):

{"items":[{"id":"garden","text":"Plant basil on Saturday and water the seedlings every morning."},{"id":"software","text":"The development server crashes when I edit a source file."}],"params":{"output_types":["dense"]}}

POST /v1/decisions (248 bytes):

{"model":"fastino/GLiNER2.5-Decide","input":"The development server crashes when I edit a source file.","questions":[{"type":"choice","name":"topic","instructions":"What is the topic?","choices":[{"value":"gardening"},{"value":"software issue"}]}]}

Request correlation:

  • Initial embedding, 16:54:11.347: 01a173fb-e9e9-7f00-907b-4a443109c2d2.
  • Initial decision refusal, 16:54:11.366: PROVISIONING, Retry-After: 60, served locally; no request ID in the captured response headers.
  • First successful decision, 16:59:20.836: 01a17400-a2f6-74c0-87c9-0e40ed0d5c47.
  • Subsequent decision, 17:02:43.375: 01a17403-b912-7c62-a02e-b4541053b455.

Related: #293 concerned workers repeatedly scaling down during loading; these models subsequently served successfully. #688/#698 already improved provisioning messages and retry hints. This report asks about the remaining Cloud first-use experience, not reopening those failures.

Lingua principale
Python
Stelle
3.5k
Fork
328
Merge medio
16h 25m
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di superlinked/sie

Tutte le issue di superlinked/sie

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.