bug: run stays EXECUTING after its task finished when the attempt completion is not recorded

Aperta
  • Qualcosa non va?
#5,060 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
node.js, redis, typescript
Ambito
backend, distributed-systems

Direzione di ricerca

Start with packages/core/src/v3/runEngineWorker/workload/http.ts, packages/redis-worker/src/worker.ts, and the run engine's heartbeatSnapshot handling; read the reproduction tests in internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts. Trace how completion failures and lost or dead-lettered heartbeat jobs leave a run executing. Done means a regression test demonstrates that the run is eventually failed and retried in the reported failure cases.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Provide environment information
  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24", with no processKeepAlive. The output above is from the machine we reproduced on.

Describe the bug

On 2026-10-24, between 10:23 and 10:27 UTC, 17 production runs finished their task code but stayed in EXECUTING for 0.5–3 hours, until we cancelled them by hand. That was about 1% of the runs that started in those minutes; the rest completed normally. The affected runs were spread over 10 different tasks and two machine sizes.

For each affected run:

  • The trace is complete. Attempt 1 > middleware() > run() all closed with no error after 0.3–62 s, and the task wrote its last log line.
  • GET /api/v3/runs/{id} returns status: "EXECUTING", durationMs: 0, costInCents: 0 and attempts: [], so the attempt completion was never recorded.
  • maxDuration (3000 s) did not end the run. As we understand it, it is enforced only while run() executes.
  • The heartbeat stall did not end the run either. The runner heartbeats from the task process, so the heartbeats stop once that process exits. The heartbeats docs say such a run fails with TASK_RUN_STALLED_EXECUTING.
  • One of the runs was a batchTriggerAndWait child, so its parent stayed waiting the whole time.

Expected: if the completion cannot be recorded, the stall handling fails the run and retries it, as the docs describe. It should not stay EXECUTING and hold a concurrency slot until someone cancels it.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/stuck-executing

To reproduce

The branch adds one run-engine test file and one standalone script, explained in repro/stuck-executing/README.md. Nothing in it is specific to our application.

1. The managed runner drops the completion after about 7.5 s

completeRunAttempt (packages/core/src/v3/runEngineWorker/workload/http.ts) uses the default wrapZodFetch retry: 5 attempts with 0.5–5 s backoff. When all of them fail, RunExecution.complete() throws, executeRun only sends the debug log failed to complete run, and the controller moves on to waitForNextRun().

repro/stuck-executing/repro.mjs runs the published RunExecution with the real WorkloadHttpClient against a fake supervisor that returns 500 for attempts/complete (or is unreachable):

npm i [email protected]   # same result with 4.5.14
node repro.mjs http500 120

+3.01s task run() finished OK; child process returns completion
+3.02s supervisor <- POST .../attempts/complete
+3.52s supervisor <- POST .../attempts/complete
+4.52s supervisor <- POST .../attempts/complete
+6.53s supervisor <- POST .../attempts/complete
+10.53s supervisor <- POST .../attempts/complete
+10.53s debugLog [execution] failed to complete run :: failed to submit completion: simulated engine failure
+10.53s debugLog [execution] [shutdown] execute call finished
(no further requests or heartbeats for 120 s)

2. The run engine can lose the heartbeatSnapshot job, and then nothing ends the run

internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts uses containerTest and a real RunEngine. Both tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/stuckExecutingRepro.test.ts --run
  • A hung completion holds the run lock. The test holds the run lock and seeds the heartbeatSnapshot job as having already failed 11 times. The next stall-handler attempt can't get the lock, which uses up the redis-worker default of 12 attempts (packages/redis-worker/src/worker.ts). The job moves to the dead-letter queue (Worker item reached max attempts. Moving to DLQ.). After the lock is released, the run stays EXECUTING with no heartbeat job, and repairEnvironment returns ignore for it.
  • The EXECUTING heartbeat enqueue is lost. The heartbeatSnapshot enqueue for the EXECUTING snapshot is dropped. Runner heartbeats keep extending the older PENDING_EXECUTING job (reschedule is ZADD XX by id). When the heartbeats stop, #handleStalledSnapshot finds the job's snapshot is "no longer the latest snapshot" and stops. The run stays EXECUTING.

We don't know which of these happened in production. Your logs and the run-engine-worker dead-letter queue would show it. We can share the run IDs privately.

Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

  • Keep the stall watchdog from dead-lettering. Give heartbeatSnapshot in workerCatalog.ts the same retry budget as ensureRunFinalized. Then, once a hung lock or an outage clears, the stall handler can still fail the run. This covers the first engine case. We put it on a branch with a regression test in heartbeats.test.ts, which fails without the change: https://github.com/triggerdotdev/trigger.dev/compare/main...PlunderStruck:trigger.dev:fix/stall-watchdog-retry. We'd be glad to open it as a PR if that's welcome.
  • Re-arm on a stale snapshot id. When #handleStalledSnapshot finds that its snapshot is no longer the latest, it could call enqueueHeartbeatIfNeeded(latestSnapshot) before returning, as checkpointSystem.ts does. This would cover the lost-enqueue case.
  • Retry the completion for longer on the runner. completeRunAttempt could use the same retry options as continueRunExecution in the same file. That would narrow the window in which a run depends on the watchdog.
Additional information
  • In our environment, completed runs per minute dropped from about 450 to about 160 at 10:24–10:25 UTC, so there may have been a short slowdown on the platform side then.
  • Related but different: #4971 and #4988, where a parent did not resume. Here the child run itself never finishes.
Lingua principale
TypeScript
Stelle
16.5k
Fork
1.5k
Merge medio
1g 15h
PR unite (30g)
17

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di triggerdotdev/trigger.dev

Tutte le issue di triggerdotdev/trigger.dev

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.