Batcher: a request that reaches an idle worker waits about 4 ms before dispatch

Abierto
#373 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
40/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
python

Línea de trabajo

Start in ModelWorker._get_next_batch_fcfs in core/worker/model_worker.py, where idle_coalesce_ms becomes coalesce_cap_ms for an idle worker, and in the 1 ms asyncio.sleep poll that the issue describes. Read WorkerConfig in core/worker/types.py and the existing idle-window tests in tests/core/test_batcher.py and tests/core/test_worker_core.py. Done means a lone sequential request shows X-Queue-Time under 1 ms while a concurrent burst still forms one batch and throughput is unchanged. The issue leaves the choice between the three proposals open, so agree the approach with maintainers first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

A request that reaches an idle model worker waits about 4 ms before its batch is dispatched. With one client sending one request at a time to an L4, X-Queue-Time is 3.8 to 4.2 ms on every request, for knowledgator/gliformer-large-v1 and knowledgator/gliclass-large-v1.0 alike. The wait is:

  • the idle accumulation window, idle_coalesce_ms (default 3.0 ms, WorkerConfig in core/worker/types.py). ModelWorker._get_next_batch_fcfs (core/worker/model_worker.py) passes it as coalesce_cap_ms when the worker was idle, so a lone request waits until arrivals have been quiet for 3 ms;
  • the idle poll, await asyncio.sleep(0.001), which checks for new requests once per millisecond.

For short requests this is a noticeable share of latency:

Workload Median per record Batcher wait Share
GLiFormer-large, one extract call per record (typed-decision records, 40 to 140 tokens) about 65 ms about 4 ms about 6%
GLiClass-large, three extract calls per record about 190 ms about 12 ms about 6%

The window exists for a reason: without it, a burst that reaches an idle worker was split into a train of small serialized forwards. The cost is that sequential and low-rate traffic, which has nothing to coalesce, pays the full window on every request.

Proposal

Keep coalescing for bursts, and stop charging it to requests that arrive alone:

  1. Dispatch at once when nothing else is arriving. When the worker is idle and only one request is pending, dispatch it immediately unless another request arrived within the last few milliseconds. The first request of a burst then runs alone and the rest coalesce behind it: two forwards instead of one, not one per request.
  2. Or adapt the window to the arrival rate. Track an exponentially weighted mean of inter-arrival gaps per model and use min(idle_coalesce_ms, k * gap) when the gap is short and 0 when it is long, so the wait follows the traffic instead of a constant.
  3. Replace the 1 ms poll with an event. Waiting on an asyncio.Event that submit() sets wakes the worker when a request arrives instead of on the next poll tick.

Setting idle_coalesce_ms: 0 already restores immediate dispatch globally, so a quick experiment can confirm the size of the saving before any code changes.

Acceptance

  • With one sequential client, X-Queue-Time on an idle worker is under 1 ms.
  • A burst of concurrent requests reaching an idle worker still forms one batch, as the existing idle-window tests (tests/core/test_batcher.py, tests/core/test_worker_core.py) check.
  • Throughput under concurrent load is unchanged within noise.
Lenguaje dominante
Python
Estrellas
3.5k
Forks
328
Merge medio
16 h 25 min
PR fusionados (30 d)
121

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de superlinked/sie

Todos los issues de superlinked/sie

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.