Cloud first-use: GLiNER2.5-Decide waits 21s before PROVISIONING; clarify readiness and latency expectations

Đang mở
#851 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
25/100
Loại issue
Tài liệu
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi

Hướng nghiên cứu

Start with the model page that shows the 13.1 ms figure and the Cloud retry docs at sie-relay#retry-while-capacity-starts and the /v1/models state section of the API reference. Reproduce the 503 PROVISIONING response from POST /v1/decisions using the bodies in the issue, and separate time-to-headers from time-to-full-body. Done means the docs state the measurement boundary and first-use expectations, and the 21-second pre-refusal wait is either explained or filed as a separate server-side investigation.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Small Cloud requests had substantially different first-use and subsequent latency. In particular, fastino/GLiNER2.5-Decide took 21.306 s before returning 503 PROVISIONING with Retry-After: 60. Later identical requests succeeded in 233–244 ms, with 7.6–10.6 ms reported inference time.

Provisioning is documented and expected. This report concerns the first-use experience: a client spends 21 seconds waiting before learning it must wait again, and a catalog entry does not make the likely wait clear. The impact is an unpredictable delay before an interactive search/classification feature can produce its first result.

Observed against https://api.superlinked.com on 2026-10-25 UTC, from one remote desktop client. Successful initial responses reported SIE version 0.10.1.

Model / tiny input Initial observed call Two subsequent calls, response fully read Subsequent X-Inference-Time
Qwen/Qwen3-Embedding-4B, 2 short texts 200, 25.192 s 563–783 ms 25.5–26.6 ms
Qwen/Qwen3-Reranker-4B, 2 pairs 200, 3.971 s 295–372 ms 28.6–28.8 ms
fastino/GLiNER2.5-Decide, 1 choice question 503, 21.306 s; retry after 60 s 200, 233–244 ms 7.6–10.6 ms
zai-org/GLM-OCR, small printed image 200, 4.000 s 287–392 ms 52.2–59.9 ms
docling, 1-page PDF 200, 11.366 s 391–419 ms 141.5–156.7 ms

The text calls started around 16:54:11; OCR/PDF around 16:55:34. The first successful decision call, at 16:59:20, took 1.441 s and reported 813.5 ms inference. The two subsequent rounds ran at 17:02:39–17:02:44, one request at a time, with no automatic retries; all 12 requests across six models returned 200. The sixth model, nvidia/parakeet-tdt-0.6b-v3, processed a valid 3.446-second clip in 370–508 ms subsequently (92.1–100 ms inference). Its earlier valid call is excluded from the first-use comparison because a previous invalid fixture had already contacted that model.

These are observations, not percentiles or a controlled cold/warm benchmark. Initial text calls ran concurrently across three different models; subsequent calls were sequential. Capacity was not forcibly evicted, other service load was not controlled, and the manual gap before the successful decision call does not measure provisioning duration. Timing headers are reported separately, not added together.

The GLiNER2.5-Decide page displays 13.1 ms per message, and qualifies the test farther down as a GPU server with the client alongside. The subsequent inference times above are compatible with that scale. The missing expectation is Cloud request latency and first-use waiting, rather than evidence that steady-state GPU execution is slow.

Before these calls, /v1/models returned state: "available", loaded: false for both embedding and decision models. That is consistent with catalog availability, but the API reference recommends using state without explaining this distinction in the model-list section.

Two concrete improvements would help:

  1. Put the measurement boundary beside the model page's headline time: colocated benchmark versus Cloud request latency, plus model-specific first-use/capacity-start expectations where known.
  2. Make first-use readiness actionable: clearly distinguish catalogued/loaded/starting states and investigate whether the 21-second wait before a retryable refusal can be shortened. Clarify whether a returned 60-second hint is a default, an estimate, or the compatibility-route cap, so a caller can present a useful waiting state.

For reproduction, send these exact bodies with normal Cloud authentication, Content-Type: application/json and Accept: application/json. Authentication is intentionally omitted. Measure time to headers and to the fully read response separately. If capacity is starting, honor Retry-After before one manual follow-up; do not force eviction or repeatedly poll inference.

POST /v1/encode/Qwen%2FQwen3-Embedding-4B (220 bytes):

{"items":[{"id":"garden","text":"Plant basil on Saturday and water the seedlings every morning."},{"id":"software","text":"The development server crashes when I edit a source file."}],"params":{"output_types":["dense"]}}

POST /v1/decisions (248 bytes):

{"model":"fastino/GLiNER2.5-Decide","input":"The development server crashes when I edit a source file.","questions":[{"type":"choice","name":"topic","instructions":"What is the topic?","choices":[{"value":"gardening"},{"value":"software issue"}]}]}

Request correlation:

  • Initial embedding, 16:54:11.347: 01a173fb-e9e9-7f00-907b-4a443109c2d2.
  • Initial decision refusal, 16:54:11.366: PROVISIONING, Retry-After: 60, served locally; no request ID in the captured response headers.
  • First successful decision, 16:59:20.836: 01a17400-a2f6-74c0-87c9-0e40ed0d5c47.
  • Subsequent decision, 17:02:43.375: 01a17403-b912-7c62-a02e-b4541053b455.

Related: #293 concerned workers repeatedly scaling down during loading; these models subsequently served successfully. #688/#698 already improved provisioning messages and retry hints. This report asks about the remaining Cloud first-use experience, not reopening those failures.

Ngôn ngữ chính
Python
Star
3.5k
Fork
328
Merge trung bình
16 giờ 25 phút
Pull request đã merge (30 ngày)
121

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của superlinked/sie

Tất cả issue của superlinked/sie

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.