Batcher: a request that reaches an idle worker waits about 4 ms before dispatch
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 40/100
- Tipo de issue
- Refactorización
- Claridad
- Bastante claro
- Estado de actividad
- Tranquilo
- Stack tecnológico
- python
- Área
- backend, performance
Línea de trabajo
Start in ModelWorker._get_next_batch_fcfs in core/worker/model_worker.py, where idle_coalesce_ms becomes coalesce_cap_ms for an idle worker, and in the 1 ms asyncio.sleep poll that the issue describes. Read WorkerConfig in core/worker/types.py and the existing idle-window tests in tests/core/test_batcher.py and tests/core/test_worker_core.py. Done means a lone sequential request shows X-Queue-Time under 1 ms while a concurrent burst still forms one batch and throughput is unchanged. The issue leaves the choice between the three proposals open, so agree the approach with maintainers first.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem
A request that reaches an idle model worker waits about 4 ms before its batch is dispatched. With one client sending one request at a time to an L4, X-Queue-Time is 3.8 to 4.2 ms on every request, for knowledgator/gliformer-large-v1 and knowledgator/gliclass-large-v1.0 alike. The wait is:
- the idle accumulation window,
idle_coalesce_ms(default 3.0 ms,WorkerConfigincore/worker/types.py).ModelWorker._get_next_batch_fcfs(core/worker/model_worker.py) passes it ascoalesce_cap_mswhen the worker was idle, so a lone request waits until arrivals have been quiet for 3 ms; - the idle poll,
await asyncio.sleep(0.001), which checks for new requests once per millisecond.
For short requests this is a noticeable share of latency:
| Workload | Median per record | Batcher wait | Share |
|---|---|---|---|
| GLiFormer-large, one extract call per record (typed-decision records, 40 to 140 tokens) | about 65 ms | about 4 ms | about 6% |
| GLiClass-large, three extract calls per record | about 190 ms | about 12 ms | about 6% |
The window exists for a reason: without it, a burst that reaches an idle worker was split into a train of small serialized forwards. The cost is that sequential and low-rate traffic, which has nothing to coalesce, pays the full window on every request.
Proposal
Keep coalescing for bursts, and stop charging it to requests that arrive alone:
- Dispatch at once when nothing else is arriving. When the worker is idle and only one request is pending, dispatch it immediately unless another request arrived within the last few milliseconds. The first request of a burst then runs alone and the rest coalesce behind it: two forwards instead of one, not one per request.
- Or adapt the window to the arrival rate. Track an exponentially weighted mean of inter-arrival gaps per model and use
min(idle_coalesce_ms, k * gap)when the gap is short and 0 when it is long, so the wait follows the traffic instead of a constant. - Replace the 1 ms poll with an event. Waiting on an
asyncio.Eventthatsubmit()sets wakes the worker when a request arrives instead of on the next poll tick.
Setting idle_coalesce_ms: 0 already restores immediate dispatch globally, so a quick experiment can confirm the size of the saving before any code changes.
Acceptance
- With one sequential client,
X-Queue-Timeon an idle worker is under 1 ms. - A burst of concurrent requests reaching an idle worker still forms one batch, as the existing idle-window tests (
tests/core/test_batcher.py,tests/core/test_worker_core.py) check. - Throughput under concurrent load is unchanged within noise.
- Lenguaje dominante
- Python
- Estrellas
- 3.5k
- Forks
- 328
- Merge medio
- 16 h 25 min
- PR fusionados (30 d)
- 121
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de superlinked/sie
-
MODEL_NOT_FOUND gives no suggestion for a model listed only as model:profilePosiblemente ocupada @ShaiShmuel la tomó hoy. Abiertoenhancement rust
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
superlinked/sie#985 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 4/5 3-5 días Aptitud para principiantes 42/100
superlinked/sie#969 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
superlinked/sie#903 · 4 comentarios ·
Los mantenedores suelen responder en 1 día
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
superlinked/sie#892 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 4/5 3-5 días Aptitud para principiantes 25/100
superlinked/sie#851 · 4 comentarios ·
Los mantenedores suelen responder en 1 día
Todos los issues de superlinked/sie
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
ip-tools/patzilla#219 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
EPSSClient commits to its SQLite cache once per score: a 100-CVE bulk fetch makes 101 disk flushesAbiertopriority:p3 technical-debt
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
jimmy058910/jmo-security-repo#1673 ·
Los mantenedores suelen responder en 1 día
-
enhancement ready
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
-
integration: frontier_silicon
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
home-assistant/core#193176 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
task create and task update take --body-file and --stdinPosiblemente ocupada @JuanBarriosS la tomó hoy. Abiertocli enhancement good first issue help wanted
Dificultad 2/5 1-3 horas Aptitud para principiantes 92/100
pyrite-wiki/pyrite#1687 · 1 comentario ·
Los mantenedores suelen responder en 1 día