Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Worker abandons work items on retryable ack failures, causing redundant activity re-execution

Abierto
  • ¿Algo no está bien?
#41 0 comentarios 0 reacciones 0 asignados Ver en GitHub

@thomcc-work ya está trabajando en esto.

Desde el 13/10/2026.

  • #42 de @pinodeca — cerrado sin fusionar
  • #72 de @thomcc-work — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
rust, sqlite
Área
backend, databases, testing-qa

Línea de trabajo

Empieza en src/runtime/dispatchers/worker.rs, en handle_activity_outcome, y compara su gestión de errores de ack con el bucle de sondeo de fetch_work_item. Después, inspecciona tests/session_e2e_tests.rs y ejecuta la prueba multi-worker indicada, incluidos los comandos de repetición proporcionados. Se considera terminado cuando los fallos de ack reintentables aplican backoff antes de abandonar, los fallos permanentes siguen provocando el abandono y la prueba coincide con el comportamiento at-least-once documentado.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Summary

A retryable ack_work_item failure causes the worker dispatcher to abandon the work item, which redelivers the activity and runs the handler again. The dispatcher poll loop already retries retryable provider errors on fetch_work_item; the ack path does not, and abandons immediately regardless of the error's retry classification.

This surfaces as a flaky test — session_e2e_tests::test_multi_worker_heterogeneous_config — but the flake is only the symptom. There are two distinct defects.

Defect 1 — worker abandons on retryable ack failures

src/runtime/dispatchers/worker.rs handle_activity_outcome abandons on any ack error:

Err(e) => {
    warn!(..., "worker: atomic ack failed, abandoning work item");
    let _ = rt.history_store
        .abandon_work_item(&ctx.lock_token, Some(Duration::from_millis(100)), false)
        .await;
    rt.record_activity_infra_error();
}

Compare the poll loop in the same file, which does branch on e.is_retryable() and backs off:

Err(e) if e.is_retryable() => {
    consecutive_retryable_errors += 1;
    let backoff_ms = (100 * 2_u64.pow(consecutive_retryable_errors)).min(3000);
    warn!("Error fetching work item (retryable, attempt {}): {:?}, backing off {}ms", ...);
    tokio::time::sleep(Duration::from_millis(backoff_ms)).await;
    continue;
}

Activities are documented as at-least-once (docs/durable-futures-internals.md, "Activity Guarantees: At-Least-Once Execution"), so re-execution is not a correctness violation. But re-running a completed handler because the database was contended for a few milliseconds is wasteful and avoidable — the ack transaction rolled back, so retrying in place is safe.

Defect 2 — test asserts exactly-once on an at-least-once runtime

tests/session_e2e_tests.rs:

assert_eq!(a + b, 3, "Total should be 3, got A={a} B={b}");

This asserts each of the 3 scheduled activities executed exactly once. Any ack failure or lock expiry redelivers, so the assertion is wrong independent of the storage layer. Observed failure: Total should be 3, got A=1 B=3.

Why the ack fails here: shared-cache SQLITE_LOCKED

SqliteProvider::new_in_memory connects with sqlite::memory:?cache=shared and max_connections(5).

Shared-cache mode replaces file-level locking with table-level read/write locks. A connection holding a read lock that tries to upgrade to a write lock while another connection still reads that table gets SQLITE_LOCKED (code 6) — reported by sqlx as database is deadlocked.

Two things follow from the SQLite shared-cache docs:

  1. PRAGMA busy_timeout = 60000 provides zero protection here. "If a required table lock cannot be obtained, the query fails and SQLITE_LOCKED is returned to the caller." No busy handler is invoked; recovering requires sqlite3_unlock_notify, which sqlx does not wire up.
  2. Writer starvation amplifies it. "After any attempt to obtain a write-lock on a table fails ... all attempts to open new transactions on the shared-cache fail until the current writer concludes." This is why fetch_work_item and fetch_orchestration_item also report SQLITE_LOCKED in the same window.

SQLite also documents shared-cache as obsolete: "Shared-cache mode is an obsolete feature. The use of shared-cache mode is discouraged."

Blast radius: shared-cache is used only by new_in_memory. File-backed deployments use WAL and get SQLITE_BUSY, which is covered by busy_timeout. So the specific SQLITE_LOCKED trigger is test-infrastructure-only — but Defect 1 applies to any provider that returns a retryable error from ack_work_item.

Reproduction

for i in $(seq 8); do cargo nt --test session_e2e_tests; done

Measured 2/8 failures in-suite. Also 1/5 in isolation:

for i in $(seq 5); do cargo nt -E 'test(test_multi_worker_heterogeneous_config)'; done

Isolation failing too is worth noting: the contention is intra-test (2 runtimes x (2 worker + 2 orchestration) dispatchers = 8 concurrent pollers against one shared cache with a 5-connection pool), not cross-test.

Failure log:

WARN duroxide::runtime::dispatchers::worker: worker: atomic ack failed,
abandoning work item instance=hetero-test execution_id=1 activity_id=3
worker_id=work-0-unconstrained-node
error=ack_work_item: error returned from database: (code: 6) database is deadlocked

Confirmed pre-existing: the assertion and the worker ack path are both unchanged at b6e0255 (origin/main).

Proposed fix

  1. Retry retryable ack failures in place with bounded exponential backoff before falling back to abandon, mirroring the existing fetch_work_item handling. Permanent errors keep abandoning immediately.
  2. Relax the test assertion to a + b >= 3, matching the documented at-least-once guarantee.
Lenguaje dominante
Rust
Estrellas
226
Forks
62
Merge medio
20 h 46 min
PR fusionados (30 d)
6

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/duroxide

Todos los issues de microsoft/duroxide

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.