Existing actor stays unreachable (no_envoys, sliding reschedule) after runner restart if woken while no envoy is connected — Wake ignored in Reallocating (2.3.27)

Read engine/packages/pegboard/src/workflows/actor2/mod.rs, especially the Reallocating transition and Main::Wake handling, then inspect the retry logic in runtime.rs and the threshold in config/pegboard.rs. Reproduce the runner restart with repeated wake requests and compare it with the quiet control. Done when an existing actor can reallocate after an envoy returns despite continuing wake traffic, with regression coverage for both cases.

Disclosure: this report and the reproduction behind it were produced by AI agents (coding agents running a validation campaign for a project built on Rivet), reviewed by an AI orchestrator, and filed with the project owner's approval. We've tried to separate what we observed from what we inferred from source. Apologies for anything we got wrong. Happy to rerun anything you'd like.

Summary

On Rivet Engine 2.3.27 in serverful (envoy) mode, if a request wakes an existing actor while no envoy is connected, the actor fails allocation with no_envoys. It then stays unreachable after the runner comes back, as long as requests keep arriving. Each new request pushes reschedule_ts further out and is not served. A fresh actor key on the same runner works immediately. The stuck actor recovers right away after POST /actors/{id}/sleep plus one request, with its durable state intact.

In our original incident, a runner restart under normal client traffic (clients reconnecting and pinging) made the actor unavailable for about 318 s. That's consistent with the default actor_retry_duration_threshold of 5 min, after which the actor goes to Sleeping and the next wake succeeds.

Environment

• rivetdev/engine:2.3.27 (Docker), single node, local filesystem/native SQLite storage (backend=LocalNative)
• rivetkit 2.3.27, serverful / native envoy mode, Node 24 runner
• One runner, runner selector default, namespace default, actor crash policy sleep
• Isolated engine with its own data directory, with no other traffic

Reproduction (confirmed)

Times are relative to the runner kill (t = 0).

1. Start the engine and one runner. Create an actor (getOrCreate by key) and write some durable state to it.
2. SIGKILL the runner and keep it down.
3. Poll the actor record (GET /actors?namespace=default&actor_ids=<id>) only, sending no requests to the actor, until it shows error = envoy_connection_lost. That happened at t ≈ 15.3 s, with sleep_ts set and reschedule_ts = null.
4. While the runner is still down (t ≈ 15.3 s), start sending requests to that existing actor through the gateway (POST /gateway/<actor>/action/ping?rvt-method=getOrCreate&rvt-key=<key>&rvt-runner=default&rvt-crash-policy=sleep). Send them one at a time with an 8 s client timeout, so requests arrive at t ≈ 15.3 s, 23.6 s and 31.9 s.
   • About 3 s later the record shows error = no_envoys and reschedule_ts ≈ t + 19.3 s.
5. Start the runner again (start issued at t ≈ 18.6 s) while requests to the existing key continue.
6. Control: a request to a fresh key at t ≈ 22.6 s → HTTP 200 in 77 ms. The runner is healthy.
7. Existing key: it stays at error = no_envoys. Across samples, reschedule_ts moved to t + 35.3 s, then t + 55.6 s (first seen right after the t ≈ 23.6 s request arrived), then t + 95.9 s (right after the t ≈ 31.9 s request). Three requests each timed out at our 8 s client timeout, with zero successes.
8. POST /actors/{id}/sleep?namespace=default → HTTP 200 in 2 ms. One request afterwards → HTTP 200 in 62 ms, the error clears and connectable_ts is set. The actor's durable state was unchanged: its single stored ledger row, pending count and cursor/epoch all matched pre-fault values.

Quiet control (confirmed)

The same steps, except no requests are sent to the actor until after the runner is back. We SIGKILLed, waited about 24 s until envoy_connection_lost, restarted the runner, then sent the first request. The actor never showed no_envoys. The first request succeeded in 64 ms with the same durable state.

So the trigger is a wake while zero envoys are available, followed by continued wake traffic. Losing the runner alone doesn't trigger it.

Observed actor-record transitions

connectable
→ error=envoy_connection_lost, sleep_ts set, reschedule_ts=null          (runner killed)
→ error=no_envoys, reschedule_ts set                                     (first wake while no runner)
→ error=no_envoys, reschedule_ts keeps sliding later                     (runner back, requests continuing)
→ error cleared, connectable_ts set                                      (POST /sleep, then one request)

The public record doesn't expose the workflow transition or generation. The Reallocating interpretation below comes from source, not from a log line we observed.

Expected vs actual

• Expected: once an envoy is registered again, the next request (or the next retry) allocates the existing actor, as it does for a fresh key and in the quiet control.
• Actual: the existing actor stays unreachable while traffic continues, for up to actor_retry_duration_threshold (5 min by default). Gateway requests time out instead of being served.

Likely mechanism (from source, v2.3.27; inferred, not instrumented)

engine/packages/pegboard/src/workflows/actor2/mod.rs:

• L604–631, Transition::Reallocating: each loop calls get_next_retry_ts(), which increments retry_count (in runtime.rs). It then does listen_n_until(next_retry_ts). reschedule_actor is only attempted if no signal arrived (if signals.is_empty()). Any incoming signal therefore cancels that retry attempt, and the next loop has a longer backoff.
• L913–931, Main::Wake during Reallocating falls into _ =>, logs "cannot wake actor that is not sleeping" and does nothing.
• L607–613: the actor goes to Sleeping only once last_retry_ts > since_ts + actor_retry_duration_threshold (config/pegboard.rs L251–253, default 5 * 60 * 1000).
• L965–968, Main::Sleep during Reallocating → Sleeping. A following Wake then calls reschedule_actor, which matches the sleep→wake workaround we observed.

So with a steady stream of wakes arriving faster than the growing backoff, the actor never attempts reallocation until the 300 s threshold.

Not yet confirmed by us: we haven't run a variant where requests stop entirely after the runner returns, to check that the actor then recovers at reschedule_ts by itself.

Hypotheses we checked and ruled out as the trigger

• SQLite close at shutdown (#5554 / PR #5860): we see failed to sync scheduled actor alarm … sqlite transaction coordinator is closed (transaction_closed) and SELECT MIN(trigger_at) FROM _rivet_schedule_events … transaction_closed when the runner stops. Fresh actors that stayed healthy logged the same error, and the quiet control recovers despite it. Our app doesn't schedule alarms, and _rivet_schedule_events was empty when we inspected it. It's real, but not the trigger here.
• #5859 (dropped event dispatch): its actor event processor failed and failed to ingest event / channel closed signatures are absent from our logs.
• #5738 (drain eviction / VFS loss): its signatures are absent, and it's a different version and setup.
• #5484 (protocol mismatch): it doesn't fit, because fresh actors work on the same runner.
• #5835 (non-hibernating WebSocket default) is already included in 2.3.27.

We found no released fix after 2.3.27, and main appears to have the same Reallocating / Wake handling.

Workaround we're using

A supervisor in our runner, after envoy registration, probes each actor with a short timeout. On a timeout or no_envoys with a future reschedule_ts, it calls POST /actors/{id}/sleep then sends one normal request. It's idempotent, with bounded retries and backoff. This preserves actor storage in our tests.

Possible fixes (suggestions only)

• In Reallocating, treat Main::Wake as "retry now", i.e. attempt reschedule_actor immediately. Alternatively, don't let an interrupting signal skip the pending retry or advance the backoff.
• And/or re-trigger reallocation for actors in Reallocating when a matching envoy registers.

We can share the repro script, with endpoints and tokens removed, if that helps.