Keep a timed-out handler visible after runWithTimeout stops waiting for it
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- typescript
- Área
- backend, observability
Línea de trabajo
Empieza en packages/outpost/queue/src/worker.ts, siguiendo runWithTimeout, processJob, activeJobStarts y la ruta de WorkerHealthStatus. Compara el seguimiento propuesto de los handlers huérfanos y el registro de timeouts con la respuesta de health y el manejo de errores existentes. Se considera terminado cuando los handlers cuyo tiempo de espera se ha agotado siguen siendo observables, health informa de orphanedHandlerCount y el worker devuelve 503 en el umbral acordado.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
runWithTimeout in packages/outpost/queue/src/worker.ts is a Promise.race between the handler and a timer. The race settling does not cancel the handler, so on a timeout the job is marked failed, processJob's finally frees the slot and deletes the job's entry from activeJobStarts — and the handler keeps running.
That matters more since #262, because overdueJobCount is measured from activeJobStarts. An orphaned handler is removed from the map at the moment it becomes orphaned, so the count is zero by construction for exactly the work that escaped its timeout. The docstring on that field says a non-zero count means the timeout machinery failed; this is a failure of the timeout machinery that it cannot see.
What it looks like in production
The LLM provider stalls, with no AbortSignal on the pipeline's fetch. Every AI_RESPONSE job times out at 120s, handleFailure schedules a retry, and the poll immediately claims more. Each bad job leaves an orphan behind and is retried up to maxAttempts, so orphans accumulate — each holding a socket and a connection-pool checkout.
Steady state after ten minutes is a worker running noticeably more work than it believes it is, while /health answers 200 {"status":"ok"} and activeJobCount reports only whatever is currently raced. The eventual symptom is pool exhaustion surfacing as an unrelated Prisma error somewhere else entirely, which is a long way from the cause.
Shape of a fix
Keep the orphan visible rather than deleting it:
const started = this.activeJobStarts.get(job.id);
void promise.finally(() => this.orphanedHandlers.delete(job.id));
this.orphanedHandlers.set(job.id, started);
Publish orphanedHandlerCount on WorkerHealthStatus and answer 503 above a threshold — >= maxConcurrency is a defensible line, since past it the worker is running more work than it thinks it is. At minimum, console.error on every timeout naming the job id and type: today the only record is the console.warn inside handleFailure, which describes a retry rather than an abandoned handler.
The deeper fix is an AbortSignal threaded through the handler contract so a timeout actually stops the work. That is a larger change to a shared package and worth doing on its own.
Pre-existing on main; #262 neither introduced nor worsened it. Filed because #262's docs now reference the limit, and a documented limit with no tracking behind it tends to become a documented permanence.
- Lenguaje dominante
- TypeScript
- Estrellas
- 13
- Forks
- 6
- Merge medio
- 4 h 49 min
- PR fusionados (30 d)
- 3
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Sin guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de CopilotKit/outpost
-
area: docs area: security roadmap: now
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
CopilotKit/outpost#277 ·
-
area: infrastructure roadmap roadmap: later
Dificultad 1/5 Menos de una hora Aptitud para principiantes 74/100
CopilotKit/outpost#179 ·
-
area: ai roadmap roadmap: now
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
CopilotKit/outpost#145 ·
-
area: integrations priority: low roadmap roadmap: later
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
CopilotKit/outpost#124 · 3 comentarios ·
-
area: integrations priority: low roadmap roadmap: later
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
CopilotKit/outpost#123 · 2 comentarios ·
Todos los issues de CopilotKit/outpost
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
paperclipai/paperclip#17151 ·
Los mantenedores suelen responder en 1 día
-
needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 83/100
AgoraIO-Extensions/agent-infra#3159 ·
Los mantenedores suelen responder en 1 día
-
blocklist addition
Dificultad 1/5 Menos de una hora Aptitud para principiantes 66/100
MetaMask/eth-phishing-detect#304596 ·
Los mantenedores suelen responder en 1 día
-
friction
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
kentcdodds/kody#4142 ·
-
[Bug] Statistics | Task statistics are always empty on MikroORM with MySQL or SQLitePosiblemente ocupada @NdekoCode la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
ever-co/ever-gauzy#11214 ·
Los mantenedores suelen responder en 1 día