bug: run stays EXECUTING after its task finished when the attempt completion is not recorded

Đang mở
  • Có gì không ổn?
#5,060 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, redis, typescript
Lĩnh vực
backend, distributed-systems

Hướng nghiên cứu

Start with packages/core/src/v3/runEngineWorker/workload/http.ts, packages/redis-worker/src/worker.ts, and the run engine's heartbeatSnapshot handling; read the reproduction tests in internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts. Trace how completion failures and lost or dead-lettered heartbeat jobs leave a run executing. Done means a regression test demonstrates that the run is eventually failed and retried in the reported failure cases.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Provide environment information
  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24", with no processKeepAlive. The output above is from the machine we reproduced on.

Describe the bug

On 2026-10-24, between 10:23 and 10:27 UTC, 17 production runs finished their task code but stayed in EXECUTING for 0.5–3 hours, until we cancelled them by hand. That was about 1% of the runs that started in those minutes; the rest completed normally. The affected runs were spread over 10 different tasks and two machine sizes.

For each affected run:

  • The trace is complete. Attempt 1 > middleware() > run() all closed with no error after 0.3–62 s, and the task wrote its last log line.
  • GET /api/v3/runs/{id} returns status: "EXECUTING", durationMs: 0, costInCents: 0 and attempts: [], so the attempt completion was never recorded.
  • maxDuration (3000 s) did not end the run. As we understand it, it is enforced only while run() executes.
  • The heartbeat stall did not end the run either. The runner heartbeats from the task process, so the heartbeats stop once that process exits. The heartbeats docs say such a run fails with TASK_RUN_STALLED_EXECUTING.
  • One of the runs was a batchTriggerAndWait child, so its parent stayed waiting the whole time.

Expected: if the completion cannot be recorded, the stall handling fails the run and retries it, as the docs describe. It should not stay EXECUTING and hold a concurrency slot until someone cancels it.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/stuck-executing

To reproduce

The branch adds one run-engine test file and one standalone script, explained in repro/stuck-executing/README.md. Nothing in it is specific to our application.

1. The managed runner drops the completion after about 7.5 s

completeRunAttempt (packages/core/src/v3/runEngineWorker/workload/http.ts) uses the default wrapZodFetch retry: 5 attempts with 0.5–5 s backoff. When all of them fail, RunExecution.complete() throws, executeRun only sends the debug log failed to complete run, and the controller moves on to waitForNextRun().

repro/stuck-executing/repro.mjs runs the published RunExecution with the real WorkloadHttpClient against a fake supervisor that returns 500 for attempts/complete (or is unreachable):

npm i [email protected]   # same result with 4.5.14
node repro.mjs http500 120

+3.01s task run() finished OK; child process returns completion
+3.02s supervisor <- POST .../attempts/complete
+3.52s supervisor <- POST .../attempts/complete
+4.52s supervisor <- POST .../attempts/complete
+6.53s supervisor <- POST .../attempts/complete
+10.53s supervisor <- POST .../attempts/complete
+10.53s debugLog [execution] failed to complete run :: failed to submit completion: simulated engine failure
+10.53s debugLog [execution] [shutdown] execute call finished
(no further requests or heartbeats for 120 s)

2. The run engine can lose the heartbeatSnapshot job, and then nothing ends the run

internal-packages/run-engine/src/engine/tests/stuckExecutingRepro.test.ts uses containerTest and a real RunEngine. Both tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/stuckExecutingRepro.test.ts --run
  • A hung completion holds the run lock. The test holds the run lock and seeds the heartbeatSnapshot job as having already failed 11 times. The next stall-handler attempt can't get the lock, which uses up the redis-worker default of 12 attempts (packages/redis-worker/src/worker.ts). The job moves to the dead-letter queue (Worker item reached max attempts. Moving to DLQ.). After the lock is released, the run stays EXECUTING with no heartbeat job, and repairEnvironment returns ignore for it.
  • The EXECUTING heartbeat enqueue is lost. The heartbeatSnapshot enqueue for the EXECUTING snapshot is dropped. Runner heartbeats keep extending the older PENDING_EXECUTING job (reschedule is ZADD XX by id). When the heartbeats stop, #handleStalledSnapshot finds the job's snapshot is "no longer the latest snapshot" and stops. The run stays EXECUTING.

We don't know which of these happened in production. Your logs and the run-engine-worker dead-letter queue would show it. We can share the run IDs privately.

Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

  • Keep the stall watchdog from dead-lettering. Give heartbeatSnapshot in workerCatalog.ts the same retry budget as ensureRunFinalized. Then, once a hung lock or an outage clears, the stall handler can still fail the run. This covers the first engine case. We put it on a branch with a regression test in heartbeats.test.ts, which fails without the change: https://github.com/triggerdotdev/trigger.dev/compare/main...PlunderStruck:trigger.dev:fix/stall-watchdog-retry. We'd be glad to open it as a PR if that's welcome.
  • Re-arm on a stale snapshot id. When #handleStalledSnapshot finds that its snapshot is no longer the latest, it could call enqueueHeartbeatIfNeeded(latestSnapshot) before returning, as checkpointSystem.ts does. This would cover the lost-enqueue case.
  • Retry the completion for longer on the runner. completeRunAttempt could use the same retry options as continueRunExecution in the same file. That would narrow the window in which a run depends on the watchdog.
Additional information
  • In our environment, completed runs per minute dropped from about 450 to about 160 at 10:24–10:25 UTC, so there may have been a short slowdown on the platform side then.
  • Related but different: #4971 and #4988, where a parent did not resume. Here the child run itself never finishes.
Ngôn ngữ chính
TypeScript
Star
16.5k
Fork
1.5k
Merge trung bình
1 ngày 15 giờ
Pull request đã merge (30 ngày)
17

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của triggerdotdev/trigger.dev

Tất cả issue của triggerdotdev/trigger.dev

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.