Expose orchestrator_lock_timeout in JsRuntimeOptions (Rust supports it, Node bindings do not)

Individua la definizione di JsRuntimeOptions e il mapping esistente di workerLockTimeoutMs, quindi confrontalo con RuntimeOptions in src/runtime/mod.rs. Segui i test esistenti delle opzioni o i controlli di build del binding; il lavoro è completato quando i consumer Node possono configurare orchestratorLockTimeoutMs e la validazione del binding ha esito positivo.

Summary

RuntimeOptions.orchestrator_lock_timeout is configurable in the Rust crate but is not surfaced through JsRuntimeOptions. Node consumers are pinned to the 5s default with no override.

The asymmetry is the tell: JsRuntimeOptions already exposes workerLockTimeoutMs → worker_lock_timeout, but its orchestrator twin was never mapped. This looks like an oversight rather than a deliberate omission.

Current state

Rust crate (duroxide 0.1.29, src/runtime/mod.rs):

/// Lock timeout for orchestrator queue items.
/// When an orchestration message is dequeued, it's locked for this duration.
/// Default: 5 seconds
pub orchestrator_lock_timeout: Duration,

Default Duration::from_secs(5), consumed in src/runtime/dispatchers/orchestration.rs, and explicitly covered by tests/lock_timeout_tests.rs (which exercises Duration::from_secs(10), from_secs(1), from_secs(60), …). The crate's own doc example even shows orchestrator_lock_timeout: Duration::from_secs(10).

Node bindings (duroxide npm 0.1.27) — JsRuntimeOptions exposes:

orchestrationConcurrency, workerConcurrency, dispatcherPollIntervalMs, workerLockTimeoutMs, logFormat, logLevel, serviceName, serviceVersion, maxSessionsPerRuntime, sessionIdleTimeoutMs, workerNodeId, workerTagFilter, workerTagFilterTags

Not mapped: orchestrator_lock_timeout, orchestrator_lock_renewal_buffer, worker_lock_renewal_buffer, dispatcher_long_poll_timeout, max_attempts, session_lock_timeout, session_lock_renewal_buffer, session_cleanup_interval, activity_cancellation_grace_period, unregistered_backoff, supported_replay_versions.

The only DUROXIDE_* environment variable in the native binary is DUROXIDE_PG_POOL_MAX, so there is no env-var escape hatch either.

Motivation: a 5-day production livelock with no available mitigation

A PilotSwarm deployment (8 worker pods × orchestrationConcurrency: 4 = 32 orchestration dispatchers against a single Postgres-backed queue) entered a self-sustaining livelock:

1. Under contention, fetch_orchestration_item began taking ~6.4s — longer than the 5s orchestrator_lock_timeout.
2. The lock token was therefore already expired by the time the turn tried to ack: ack_orchestration_item: Invalid lock token.
3. Because the ack failed, retry bookkeeping never committed — including the poison-marking after max_attempts. The runtime logged Orchestration message exceeded max attempts, marking as poison on every single pass, but the marking could never land.
4. With no backoff committed either, the row's lock simply expired after 5s and the message became visible again. The effective retry interval was pinned at exactly the lock lease.
5. A permanently non-empty queue kept all 32 dispatchers in constant contention — which is precisely what made the fetch slow in step 1.

The loop is stable rather than self-correcting, because step 5 causes step 1.

Observed at the point of intervention:

Metric	Value
Max attempts on a single instance	78,987 (against max_attempts: 10)
Total accumulated attempts	781,446 across 12 instances
fetch_orchestration_item latency	~6.4s, very tightly clustered
Effective throughput	zero
Duration before manual intervention	5 days

Latency clustered extremely tightly (6.40, 6.39, 6.39, 6.26, 6.27, 6.45s), which is the signature of queueing/serialization rather than scan cost.

Note that lock renewal cannot help here. The documented strategy renews at 0.5 × timeout when orchestrator_lock_timeout < 15s (so 2.5s at the default), but the 6.4s is spent inside fetch_orchestration_item — before any item exists to renew.

Recovery required manually deleting the affected orchestration instances out-of-band. With orchestrator_lock_timeout exposed, raising it to 10s would have let the ack — and therefore the poison-marking — commit, allowing the runtime's own poison handling to drain the queue without operator intervention.

Ask

Map orchestrator_lock_timeout into JsRuntimeOptions as orchestratorLockTimeoutMs, exactly as workerLockTimeoutMs already maps to worker_lock_timeout.

Ideally also expose orchestratorLockRenewalBufferMs and maxAttempts, which are part of the same tuning surface.

Secondary observation: default divergence

Two values differ notably between the Rust defaults and what Node consumers run, and they compound the contention that triggers this:

Option	Rust default	Observed in Node consumers
dispatcher_min_poll_interval	100ms	10ms
orchestration_concurrency	2	4 (× N pods)

At 32 dispatchers polling every 10ms, that is up to ~3,200 fetch_orchestration_item calls/sec against one queue table.

It would be worth documenting the interaction explicitly: total dispatchers × poll frequency determines contention on the orchestrator queue, and orchestrator_lock_timeout must exceed worst-case fetch_orchestration_item latency under that load, or the system cannot make forward progress.

Related

The underlying reason a too-short lease is unrecoverable rather than merely slow — the poison-marking path depending on a lock token that has already expired — is filed separately against the Rust crate.

Environment

• duroxide (npm) 0.1.27
• duroxide (crate) 0.1.29
• Postgres-backed provider