Cloud first-use: GLiNER2.5-Decide waits 21s before PROVISIONING; clarify readiness and latency expectations
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 25/100
- Tipo di issue
- Documentazione
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Ambito
- api, cloud, documentation, performance
Direzione di ricerca
Start with the model page that shows the 13.1 ms figure and the Cloud retry docs at sie-relay#retry-while-capacity-starts and the /v1/models state section of the API reference. Reproduce the 503 PROVISIONING response from POST /v1/decisions using the bodies in the issue, and separate time-to-headers from time-to-full-body. Done means the docs state the measurement boundary and first-use expectations, and the 21-second pre-refusal wait is either explained or filed as a separate server-side investigation.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Small Cloud requests had substantially different first-use and subsequent latency. In particular, fastino/GLiNER2.5-Decide took 21.306 s before returning 503 PROVISIONING with Retry-After: 60. Later identical requests succeeded in 233–244 ms, with 7.6–10.6 ms reported inference time.
Provisioning is documented and expected. This report concerns the first-use experience: a client spends 21 seconds waiting before learning it must wait again, and a catalog entry does not make the likely wait clear. The impact is an unpredictable delay before an interactive search/classification feature can produce its first result.
Observed against https://api.superlinked.com on 2026-10-25 UTC, from one remote desktop client. Successful initial responses reported SIE version 0.10.1.
| Model / tiny input | Initial observed call | Two subsequent calls, response fully read | Subsequent X-Inference-Time |
|---|---|---|---|
Qwen/Qwen3-Embedding-4B, 2 short texts |
200, 25.192 s | 563–783 ms | 25.5–26.6 ms |
Qwen/Qwen3-Reranker-4B, 2 pairs |
200, 3.971 s | 295–372 ms | 28.6–28.8 ms |
fastino/GLiNER2.5-Decide, 1 choice question |
503, 21.306 s; retry after 60 s | 200, 233–244 ms | 7.6–10.6 ms |
zai-org/GLM-OCR, small printed image |
200, 4.000 s | 287–392 ms | 52.2–59.9 ms |
docling, 1-page PDF |
200, 11.366 s | 391–419 ms | 141.5–156.7 ms |
The text calls started around 16:54:11; OCR/PDF around 16:55:34. The first successful decision call, at 16:59:20, took 1.441 s and reported 813.5 ms inference. The two subsequent rounds ran at 17:02:39–17:02:44, one request at a time, with no automatic retries; all 12 requests across six models returned 200. The sixth model, nvidia/parakeet-tdt-0.6b-v3, processed a valid 3.446-second clip in 370–508 ms subsequently (92.1–100 ms inference). Its earlier valid call is excluded from the first-use comparison because a previous invalid fixture had already contacted that model.
These are observations, not percentiles or a controlled cold/warm benchmark. Initial text calls ran concurrently across three different models; subsequent calls were sequential. Capacity was not forcibly evicted, other service load was not controlled, and the manual gap before the successful decision call does not measure provisioning duration. Timing headers are reported separately, not added together.
The GLiNER2.5-Decide page displays 13.1 ms per message, and qualifies the test farther down as a GPU server with the client alongside. The subsequent inference times above are compatible with that scale. The missing expectation is Cloud request latency and first-use waiting, rather than evidence that steady-state GPU execution is slow.
Before these calls, /v1/models returned state: "available", loaded: false for both embedding and decision models. That is consistent with catalog availability, but the API reference recommends using state without explaining this distinction in the model-list section.
Two concrete improvements would help:
- Put the measurement boundary beside the model page's headline time: colocated benchmark versus Cloud request latency, plus model-specific first-use/capacity-start expectations where known.
- Make first-use readiness actionable: clearly distinguish catalogued/loaded/starting states and investigate whether the 21-second wait before a retryable refusal can be shortened. Clarify whether a returned 60-second hint is a default, an estimate, or the compatibility-route cap, so a caller can present a useful waiting state.
For reproduction, send these exact bodies with normal Cloud authentication, Content-Type: application/json and Accept: application/json. Authentication is intentionally omitted. Measure time to headers and to the fully read response separately. If capacity is starting, honor Retry-After before one manual follow-up; do not force eviction or repeatedly poll inference.
POST /v1/encode/Qwen%2FQwen3-Embedding-4B (220 bytes):
{"items":[{"id":"garden","text":"Plant basil on Saturday and water the seedlings every morning."},{"id":"software","text":"The development server crashes when I edit a source file."}],"params":{"output_types":["dense"]}}
POST /v1/decisions (248 bytes):
{"model":"fastino/GLiNER2.5-Decide","input":"The development server crashes when I edit a source file.","questions":[{"type":"choice","name":"topic","instructions":"What is the topic?","choices":[{"value":"gardening"},{"value":"software issue"}]}]}
Request correlation:
- Initial embedding, 16:54:11.347:
01a173fb-e9e9-7f00-907b-4a443109c2d2. - Initial decision refusal, 16:54:11.366:
PROVISIONING,Retry-After: 60, served locally; no request ID in the captured response headers. - First successful decision, 16:59:20.836:
01a17400-a2f6-74c0-87c9-0e40ed0d5c47. - Subsequent decision, 17:02:43.375:
01a17403-b912-7c62-a02e-b4541053b455.
Related: #293 concerned workers repeatedly scaling down during loading; these models subsequently served successfully. #688/#698 already improved provisioning messages and retry hints. This report asks about the remaining Cloud first-use experience, not reopening those failures.
- Lingua principale
- Python
- Stelle
- 3.5k
- Fork
- 328
- Merge medio
- 16h 25m
- PR unite (30g)
- 121
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di superlinked/sie
-
MODEL_NOT_FOUND gives no suggestion for a model listed only as model:profileForse già presa @ShaiShmuel l’ha presa oggi. Apertaenhancement rust
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
superlinked/sie#985 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 42/100
superlinked/sie#969 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
superlinked/sie#903 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
superlinked/sie#892 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 20/100
superlinked/sie#840 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di superlinked/sie
Issue simili
-
Difficoltà 1/5 1-3 ore Idoneità per principianti 76/100
samuelclay/NewsBlur#2214 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
readthedocs/readthedocs-builder#73 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
reflex-dev/reflex#7915 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno