bug: a resumed run whose restore fails is requeued behind newer runs

Đang mở
  • Có gì không ổn?
#5,104 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

  • #5114 của @granbellllc-png — đã đóng, không merge

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, typescript
Lĩnh vực
backend, devtools, distributed-systems

Hướng nghiên cứu

Start with internal-packages/run-engine/src/run-queue/index.ts and trace #callNackMessage alongside tryNackAndRequeue, which handles checkpointed runs. Read the reproduction instructions and tests in the linked repro/resumed-run-queue-position branch, then compare the suggested fix with the direct nack paths in dequeueSystem.ts and engine/index.ts. Done means a regression test shows a failed restore keeps its queue position ahead of newer runs while repeated failures still back off.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Provide environment information
  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with the trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24". The output above is from the machine we reproduced on.

Describe the bug

A run that resumes after a wait is enqueued at its original queue position. continueRunIfUnblocked says so: "Re-enqueue with the original timestamp, so waiting runs dequeue before new runs." If the restore then fails before the run continues, the run is requeued through tryNackAndRequeue with no retry time. RunQueue then scores the message at now + backoff (internal-packages/run-engine/src/run-queue/index.ts, #callNackMessage). The run loses its position and waits behind every run that was triggered after it.

What we saw on 2026-11-06:

  • Two parent runs called batchTriggerAndWait with about 50 children each, and were checkpointed while they waited. Their queue has concurrencyLimit: 4 and held a backlog of about 100 queued runs.
  • All children ended by 21:03 and 21:59 UTC. A few children were OOM-killed, and each of those was marked CRASHED (TASK_PROCESS_OOM_KILLED) 300 s after it started.
  • Each parent was then requeued once, at 22:21 and 22:04 UTC. After that the API showed status: "QUEUED", attempts: [], and a trace that ends at batchTriggerAndWait(). In the dashboard the run looks as if it never started.
  • The parents stayed queued for over 70 minutes while runs created after them kept starting in the same queue. We cancelled them at 23:32 UTC. That was about 4.5 minutes before the last run that was queued ahead of them started, so we think they were waiting their turn, not lost.
  • Each requeue freed a queue slot that another run took within 0.2 s. As far as we can tell from the code, only tryNackAndRequeue writes PENDING for a run that is resuming from a checkpoint.

Expected: a resume whose restore has to be retried keeps its place ahead of runs triggered after it, as the first resume does.

We can't see why the restores failed. The parents ran on a small machine and were close to its memory limit when they started waiting, so we think the restored process was OOM-killed. That part is our sizing. See "Additional information" for how such a restore is reported.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/resumed-run-queue-position

To reproduce

The branch adds one run-engine test file, explained in repro/resumed-run-queue-position/README.md. It uses containerTest and a real RunEngine. The tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/resumedRunQueuePositionRepro.test.ts --run

Each test:

  1. Triggers a parent in a queue with concurrency limit 1, starts it, and blocks it on a waitpoint.
  2. Checkpoints it, then completes the waitpoint, so the parent is enqueued for a restore and dequeued ahead of newer runs, as designed.
  3. Triggers a newer run while the restore holds the slot.
  4. Fails the restore before continueRunExecution, in one of two ways:
    • Stall. The restored runner never continues, so the PENDING_EXECUTING heartbeat requeues it.
    • Supervisor report. The supervisor reports the restore as requeue, as classifyRestoreFailure does for a Runner that failed with reason OOMKilled.
  5. Asserts that the parent is PENDING with its checkpoint, that the next dequeue returns the newer run, and that the parent is dequeued only after the newer run finishes.
Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

  • Keep the position on the first requeue of a resume. tryNackAndRequeue already knows whether the run has a checkpoint. It could ask nackMessage to redeliver at the message's original score, as the resume enqueue does. Later requeues would keep the backoff, so a restore that keeps failing does not spend the 12 queue attempts in a tight loop. Cold starts would be unchanged. We put this on a branch with a regression test in restoreOutcome.test.ts, which fails without the change: https://github.com/triggerdotdev/trigger.dev/compare/main...PlunderStruck:trigger.dev:fix/resumed-run-queue-position. It adds 18 lines to runAttemptSystem.ts and run-queue/index.ts, most of them comments. We'd be glad to open it as a PR if that's welcome.
  • Not covered by that branch: the direct runQueue.nackMessage calls in dequeueSystem.ts and engine/index.ts (error and admin paths) still requeue a resume at now + backoff.
Additional information
  • An OOM-killed restore is retried on the same machine with no reason recorded. classifyRestoreFailure (apps/supervisor/src/workloadManager/runCrd.ts) maps OOMKilled to its default { outcome: "requeue" }. The runner-kill reporter also skips a run whose snapshot is not EXECUTING. The run therefore records no attempt and no error, retry.outOfMemory can't move it to a larger machine, and each retry goes to the back of the queue until the 12 queue attempts are spent. Mapping OOMKilled to a failure with TASK_PROCESS_OOM_KILLED would let retry.outOfMemory apply. We haven't changed this, because we can't confirm the restore reason on Cloud.
  • No trace or attempt event for a requeue. tryNackAndRequeue emits no run event. A run that has resumed and been requeued looks the same as one that has never started.
  • Related but different: #4971 and #4988 (the parent was never resumed) and #5060 (the run stayed EXECUTING after its task finished). Here the parent is resumed but then requeued behind newer runs.
  • We can share the run IDs privately.
Ngôn ngữ chính
TypeScript
Star
16.5k
Fork
1.5k
Merge trung bình
1 ngày 15 giờ
Pull request đã merge (30 ngày)
17

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của triggerdotdev/trigger.dev

Tất cả issue của triggerdotdev/trigger.dev

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.