bug: a resumed run whose restore fails is requeued behind newer runs

Abierto
  • ¿Algo no está bien?
#5,104 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

  • #5114 de @granbellllc-png — cerrado sin fusionar

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
node.js, typescript
Área
backend, devtools, distributed-systems

Línea de trabajo

Start with internal-packages/run-engine/src/run-queue/index.ts and trace #callNackMessage alongside tryNackAndRequeue, which handles checkpointed runs. Read the reproduction instructions and tests in the linked repro/resumed-run-queue-position branch, then compare the suggested fix with the direct nack paths in dequeueSystem.ts and engine/index.ts. Done means a regression test shows a failed restore keeps its queue position ahead of newer runs while repeated failures still back off.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Provide environment information
  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with the trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24". The output above is from the machine we reproduced on.

Describe the bug

A run that resumes after a wait is enqueued at its original queue position. continueRunIfUnblocked says so: "Re-enqueue with the original timestamp, so waiting runs dequeue before new runs." If the restore then fails before the run continues, the run is requeued through tryNackAndRequeue with no retry time. RunQueue then scores the message at now + backoff (internal-packages/run-engine/src/run-queue/index.ts, #callNackMessage). The run loses its position and waits behind every run that was triggered after it.

What we saw on 2026-11-06:

  • Two parent runs called batchTriggerAndWait with about 50 children each, and were checkpointed while they waited. Their queue has concurrencyLimit: 4 and held a backlog of about 100 queued runs.
  • All children ended by 21:03 and 21:59 UTC. A few children were OOM-killed, and each of those was marked CRASHED (TASK_PROCESS_OOM_KILLED) 300 s after it started.
  • Each parent was then requeued once, at 22:21 and 22:04 UTC. After that the API showed status: "QUEUED", attempts: [], and a trace that ends at batchTriggerAndWait(). In the dashboard the run looks as if it never started.
  • The parents stayed queued for over 70 minutes while runs created after them kept starting in the same queue. We cancelled them at 23:32 UTC. That was about 4.5 minutes before the last run that was queued ahead of them started, so we think they were waiting their turn, not lost.
  • Each requeue freed a queue slot that another run took within 0.2 s. As far as we can tell from the code, only tryNackAndRequeue writes PENDING for a run that is resuming from a checkpoint.

Expected: a resume whose restore has to be retried keeps its place ahead of runs triggered after it, as the first resume does.

We can't see why the restores failed. The parents ran on a small machine and were close to its memory limit when they started waiting, so we think the restored process was OOM-killed. That part is our sizing. See "Additional information" for how such a restore is reported.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/resumed-run-queue-position

To reproduce

The branch adds one run-engine test file, explained in repro/resumed-run-queue-position/README.md. It uses containerTest and a real RunEngine. The tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/resumedRunQueuePositionRepro.test.ts --run

Each test:

  1. Triggers a parent in a queue with concurrency limit 1, starts it, and blocks it on a waitpoint.
  2. Checkpoints it, then completes the waitpoint, so the parent is enqueued for a restore and dequeued ahead of newer runs, as designed.
  3. Triggers a newer run while the restore holds the slot.
  4. Fails the restore before continueRunExecution, in one of two ways:
    • Stall. The restored runner never continues, so the PENDING_EXECUTING heartbeat requeues it.
    • Supervisor report. The supervisor reports the restore as requeue, as classifyRestoreFailure does for a Runner that failed with reason OOMKilled.
  5. Asserts that the parent is PENDING with its checkpoint, that the next dequeue returns the newer run, and that the parent is dequeued only after the newer run finishes.
Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

  • Keep the position on the first requeue of a resume. tryNackAndRequeue already knows whether the run has a checkpoint. It could ask nackMessage to redeliver at the message's original score, as the resume enqueue does. Later requeues would keep the backoff, so a restore that keeps failing does not spend the 12 queue attempts in a tight loop. Cold starts would be unchanged. We put this on a branch with a regression test in restoreOutcome.test.ts, which fails without the change: https://github.com/triggerdotdev/trigger.dev/compare/main...PlunderStruck:trigger.dev:fix/resumed-run-queue-position. It adds 18 lines to runAttemptSystem.ts and run-queue/index.ts, most of them comments. We'd be glad to open it as a PR if that's welcome.
  • Not covered by that branch: the direct runQueue.nackMessage calls in dequeueSystem.ts and engine/index.ts (error and admin paths) still requeue a resume at now + backoff.
Additional information
  • An OOM-killed restore is retried on the same machine with no reason recorded. classifyRestoreFailure (apps/supervisor/src/workloadManager/runCrd.ts) maps OOMKilled to its default { outcome: "requeue" }. The runner-kill reporter also skips a run whose snapshot is not EXECUTING. The run therefore records no attempt and no error, retry.outOfMemory can't move it to a larger machine, and each retry goes to the back of the queue until the 12 queue attempts are spent. Mapping OOMKilled to a failure with TASK_PROCESS_OOM_KILLED would let retry.outOfMemory apply. We haven't changed this, because we can't confirm the restore reason on Cloud.
  • No trace or attempt event for a requeue. tryNackAndRequeue emits no run event. A run that has resumed and been requeued looks the same as one that has never started.
  • Related but different: #4971 and #4988 (the parent was never resumed) and #5060 (the run stayed EXECUTING after its task finished). Here the parent is resumed but then requeued behind newer runs.
  • We can share the run IDs privately.
Lenguaje dominante
TypeScript
Estrellas
16.5k
Forks
1.5k
Merge medio
1 d 15 h
PR fusionados (30 d)
17

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de triggerdotdev/trigger.dev

Todos los issues de triggerdotdev/trigger.dev

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.