Batcher: a request that reaches an idle worker waits about 4 ms before dispatch
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 40/100
- Tipo di issue
- Refactoring
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- python
- Ambito
- backend, performance
Direzione di ricerca
Start in ModelWorker._get_next_batch_fcfs in core/worker/model_worker.py, where idle_coalesce_ms becomes coalesce_cap_ms for an idle worker, and in the 1 ms asyncio.sleep poll that the issue describes. Read WorkerConfig in core/worker/types.py and the existing idle-window tests in tests/core/test_batcher.py and tests/core/test_worker_core.py. Done means a lone sequential request shows X-Queue-Time under 1 ms while a concurrent burst still forms one batch and throughput is unchanged. The issue leaves the choice between the three proposals open, so agree the approach with maintainers first.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem
A request that reaches an idle model worker waits about 4 ms before its batch is dispatched. With one client sending one request at a time to an L4, X-Queue-Time is 3.8 to 4.2 ms on every request, for knowledgator/gliformer-large-v1 and knowledgator/gliclass-large-v1.0 alike. The wait is:
- the idle accumulation window,
idle_coalesce_ms(default 3.0 ms,WorkerConfigincore/worker/types.py).ModelWorker._get_next_batch_fcfs(core/worker/model_worker.py) passes it ascoalesce_cap_mswhen the worker was idle, so a lone request waits until arrivals have been quiet for 3 ms; - the idle poll,
await asyncio.sleep(0.001), which checks for new requests once per millisecond.
For short requests this is a noticeable share of latency:
| Workload | Median per record | Batcher wait | Share |
|---|---|---|---|
| GLiFormer-large, one extract call per record (typed-decision records, 40 to 140 tokens) | about 65 ms | about 4 ms | about 6% |
| GLiClass-large, three extract calls per record | about 190 ms | about 12 ms | about 6% |
The window exists for a reason: without it, a burst that reaches an idle worker was split into a train of small serialized forwards. The cost is that sequential and low-rate traffic, which has nothing to coalesce, pays the full window on every request.
Proposal
Keep coalescing for bursts, and stop charging it to requests that arrive alone:
- Dispatch at once when nothing else is arriving. When the worker is idle and only one request is pending, dispatch it immediately unless another request arrived within the last few milliseconds. The first request of a burst then runs alone and the rest coalesce behind it: two forwards instead of one, not one per request.
- Or adapt the window to the arrival rate. Track an exponentially weighted mean of inter-arrival gaps per model and use
min(idle_coalesce_ms, k * gap)when the gap is short and 0 when it is long, so the wait follows the traffic instead of a constant. - Replace the 1 ms poll with an event. Waiting on an
asyncio.Eventthatsubmit()sets wakes the worker when a request arrives instead of on the next poll tick.
Setting idle_coalesce_ms: 0 already restores immediate dispatch globally, so a quick experiment can confirm the size of the saving before any code changes.
Acceptance
- With one sequential client,
X-Queue-Timeon an idle worker is under 1 ms. - A burst of concurrent requests reaching an idle worker still forms one batch, as the existing idle-window tests (
tests/core/test_batcher.py,tests/core/test_worker_core.py) check. - Throughput under concurrent load is unchanged within noise.
- Lingua principale
- Python
- Stelle
- 3.5k
- Fork
- 328
- Merge medio
- 16h 25m
- PR unite (30g)
- 121
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di superlinked/sie
-
MODEL_NOT_FOUND gives no suggestion for a model listed only as model:profileForse già presa @ShaiShmuel l’ha presa oggi. Apertaenhancement rust
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
superlinked/sie#985 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 42/100
superlinked/sie#969 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
superlinked/sie#903 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
superlinked/sie#892 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
superlinked/sie#851 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di superlinked/sie
Issue simili
-
Difficoltà 1/5 1-3 ore Idoneità per principianti 76/100
samuelclay/NewsBlur#2214 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
readthedocs/readthedocs-builder#73 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
reflex-dev/reflex#7915 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno