Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

v2 (envoy) actors ignore `crash_policy`: a lost runner parks the actor (Sleep) instead of rescheduling it

Aperta
  • Qualcosa non va?
#5,908 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
node.js, postgresql, rust, typescript
Ambito
api, backend, distributed-systems

Direzione di ricerca

Start by reading pegboard/src/workflows/actor2/runtime.rs and pegboard/src/ops/actor/util.rs, where the issue identifies the lost-runner decision and reported crash policy. Compare that behavior with versions.mdx and actor-rescheduling.md, then check whether existing tests cover v2 actor rescheduling and crash policy. Done means crash_policy: "restart" is honored after a runner loss and the documented behavior matches; the issue also raises exposing this option in the rivetkit client.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Environment
  • rivetkit 2.3.27, rivetdev/engine:2.3.27 (self-hosted, Docker), Postgres backend
  • Node 22, one runner pool (default), actors created through rivetkit and through the engine REST API
Expected

versions.mdx says that on SIGKILL "actors are rescheduled with a crash backoff penalty", and
actor-rescheduling.md maps a lost runner to reschedule_actor. Creating an actor with
crash_policy: "restart" should make it restart on another (or the restarted) runner after the
runner is lost.

Actual

After kill -9 of the runner process, the actor's last stop reports crash_policy: "sleep",
error: {envoy_connection_lost}, and the actor stays parked until something sends it a message or
an alarm fires. A workflow in progress does not resume on its own.

  • rivetkit's client always sends "sleep" and does not expose the option.
  • Creating the actor via the engine REST API with crash_policy: "restart" still reports "sleep".
  • Engine source (2.3.27 and main): pegboard/src/workflows/actor2/runtime.rs handle_stopped
    maps StoppedVariant::Lost => Decision::Sleep and only reallocates if an alarm is due;
    pegboard/src/ops/actor/util.rs reports v2 actors as CrashPolicy::Sleep unconditionally.

Related: #3725 (closed, not planned), PR #4822 "restore crash_policy plumbing" (closed unmerged),
#5894 (open; wake ignored while Reallocating — different trigger).

Repro
// runner.ts — start with `tsx runner.ts`, then `kill -9` it while `work` is mid-loop, restart it.
import { actor, setup } from "rivetkit";
import { workflow } from "rivetkit/workflow";

const job = actor({
  run: workflow(async (ctx) => {
    await ctx.queue.next("go");
    for (let i = 0; i < 20; i++) {
      await ctx.step(`page-${i}`, async () => {
        await new Promise((r) => setTimeout(r, 500));
        console.log("page", i, "done");
      });
    }
  }),
  actions: { go: async (c) => c.queue.send("go", {}) },
});
const registry = setup({ use: { job }, endpoint: "http://127.0.0.1:6420", token: "<admin-token>", namespace: "default" });
await registry.startAndWait();
  1. Start the engine (2.3.27) and runner.ts; call client.job.getOrCreate(["a"]).go().
  2. After ~4 pages, kill -9 the runner; restart it 3 s later.
  3. Observe: no further page N done lines. The actor stays asleep (GET /actors/{id} shows
    crash_policy: "sleep", envoy_connection_lost) until it is messaged.
  4. Same result when the actor is created with POST /actors and "crash_policy": "restart".
Measured
  • Stock 2.3.27, 3 s runner downtime: the workflow did not finish within 300 s (2 of 2 runs, with
    and without crash_policy: "restart" on the REST create).
  • Workaround we ship: a durable alarm (c.run.setWakeAt, 3 s, refreshed every 1 s, cleared in
    onSleep) plus pegboard.envoy_lost_threshold = 6 s (default 15 s). Resume after restart:
    3.4–8.5 s over 4 spike runs; 2.6–2.7 s in our integration suite (3 runs); 0 completed steps re-run.
  • SIGTERM drain (rolling deploy) is fine without any workaround: resume 0.4–0.7 s.
Question / suggested fix

Is Lost => Sleep for v2 actors intended? If yes, please fix versions.mdx /
actor-rescheduling.md. If not, honour crash_policy: "restart" for v2 actors (and expose it in
rivetkit actor options) so users don't need an alarm-based watchdog to survive a runner crash.

Lingua principale
Rust
Stelle
6.3k
Fork
280
Merge medio
22h 56m
PR unite (30g)
41

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di rivet-dev/rivet

Tutte le issue di rivet-dev/rivet

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.