bug: run stays EXECUTING after its task finished when the attempt completion is not recorded

Aberta
  • Algo errado?
#5,060 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
42/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
node.js, redis, typescript
Domínio
backend, distributed-systems

Direção de pesquisa

Start with packages/core/src/v3/runEngineWorker/workload/http.ts, packages/redis-worker/src/worker.ts, and the run engine's heartbeatSnapshot handling; read the reproduction tests in internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts. Trace how completion failures and lost or dead-lettered heartbeat jobs leave a run executing. Done means a regression test demonstrates that the run is eventually failed and retried in the reported failure cases.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Provide environment information
  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24", with no processKeepAlive. The output above is from the machine we reproduced on.

Describe the bug

On 2026-10-24, between 10:23 and 10:27 UTC, 17 production runs finished their task code but stayed in EXECUTING for 0.5–3 hours, until we cancelled them by hand. That was about 1% of the runs that started in those minutes; the rest completed normally. The affected runs were spread over 10 different tasks and two machine sizes.

For each affected run:

  • The trace is complete. Attempt 1 > middleware() > run() all closed with no error after 0.3–62 s, and the task wrote its last log line.
  • GET /api/v3/runs/{id} returns status: "EXECUTING", durationMs: 0, costInCents: 0 and attempts: [], so the attempt completion was never recorded.
  • maxDuration (3000 s) did not end the run. As we understand it, it is enforced only while run() executes.
  • The heartbeat stall did not end the run either. The runner heartbeats from the task process, so the heartbeats stop once that process exits. The heartbeats docs say such a run fails with TASK_RUN_STALLED_EXECUTING.
  • One of the runs was a batchTriggerAndWait child, so its parent stayed waiting the whole time.

Expected: if the completion cannot be recorded, the stall handling fails the run and retries it, as the docs describe. It should not stay EXECUTING and hold a concurrency slot until someone cancels it.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/stuck-executing

To reproduce

The branch adds one run-engine test file and one standalone script, explained in repro/stuck-executing/README.md. Nothing in it is specific to our application.

1. The managed runner drops the completion after about 7.5 s

completeRunAttempt (packages/core/src/v3/runEngineWorker/workload/http.ts) uses the default wrapZodFetch retry: 5 attempts with 0.5–5 s backoff. When all of them fail, RunExecution.complete() throws, executeRun only sends the debug log failed to complete run, and the controller moves on to waitForNextRun().

repro/stuck-executing/repro.mjs runs the published RunExecution with the real WorkloadHttpClient against a fake supervisor that returns 500 for attempts/complete (or is unreachable):

npm i [email protected]   # same result with 4.5.14
node repro.mjs http500 120

+3.01s task run() finished OK; child process returns completion
+3.02s supervisor <- POST .../attempts/complete
+3.52s supervisor <- POST .../attempts/complete
+4.52s supervisor <- POST .../attempts/complete
+6.53s supervisor <- POST .../attempts/complete
+10.53s supervisor <- POST .../attempts/complete
+10.53s debugLog [execution] failed to complete run :: failed to submit completion: simulated engine failure
+10.53s debugLog [execution] [shutdown] execute call finished
(no further requests or heartbeats for 120 s)

2. The run engine can lose the heartbeatSnapshot job, and then nothing ends the run

internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts uses containerTest and a real RunEngine. Both tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/stuckExecutingRepro.test.ts --run
  • A hung completion holds the run lock. The test holds the run lock and seeds the heartbeatSnapshot job as having already failed 11 times. The next stall-handler attempt can't get the lock, which uses up the redis-worker default of 12 attempts (packages/redis-worker/src/worker.ts). The job moves to the dead-letter queue (Worker item reached max attempts. Moving to DLQ.). After the lock is released, the run stays EXECUTING with no heartbeat job, and repairEnvironment returns ignore for it.
  • The EXECUTING heartbeat enqueue is lost. The heartbeatSnapshot enqueue for the EXECUTING snapshot is dropped. Runner heartbeats keep extending the older PENDING_EXECUTING job (reschedule is ZADD XX by id). When the heartbeats stop, #handleStalledSnapshot finds the job's snapshot is "no longer the latest snapshot" and stops. The run stays EXECUTING.

We don't know which of these happened in production. Your logs and the run-engine-worker dead-letter queue would show it. We can share the run IDs privately.

Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

  • Keep the stall watchdog from dead-lettering. Give heartbeatSnapshot in workerCatalog.ts the same retry budget as ensureRunFinalized. Then, once a hung lock or an outage clears, the stall handler can still fail the run. This covers the first engine case. We put it on a branch with a regression test in heartbeats.test.ts, which fails without the change: https://github.com/triggerdotdev/trigger.dev/compare/main...PlunderStruck:trigger.dev:fix/stall-watchdog-retry. We'd be glad to open it as a PR if that's welcome.
  • Re-arm on a stale snapshot id. When #handleStalledSnapshot finds that its snapshot is no longer the latest, it could call enqueueHeartbeatIfNeeded(latestSnapshot) before returning, as checkpointSystem.ts does. This would cover the lost-enqueue case.
  • Retry the completion for longer on the runner. completeRunAttempt could use the same retry options as continueRunExecution in the same file. That would narrow the window in which a run depends on the watchdog.
Additional information
  • In our environment, completed runs per minute dropped from about 450 to about 160 at 10:24–10:25 UTC, so there may have been a short slowdown on the platform side then.
  • Related but different: #4971 and #4988, where a parent did not resume. Here the child run itself never finishes.
Linguagem predominante
TypeScript
Estrelas
16.5k
Forks
1.5k
Merge médio
1d 15h
PRs com merge (30d)
17

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de triggerdotdev/trigger.dev

Todas as issues de triggerdotdev/trigger.dev

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.