Provider validation: lease fencing and exactly-once consumption

Abierto
  • ¿Algo no está bien?
#75 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
postgresql, rust, sqlite
Área
backend, testing

Línea de trabajo

Start with the Provider trait in src/providers/mod.rs and the lock expiration tests, especially lock_expiration and instance_locking. Read the provider implementation guide at docs/provider-implementation-guide.md to confirm the lease renewal contract, including whether renewal can shorten a lease. Add the listed provider trait tests and verify them on SQLite; acceptance also calls for checking the PostgreSQL provider and measuring test 6's failure rate on its earlier revision.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Why

microsoft/duroxide-pg#44 fixes a provider bug that the validation suite did not catch, microsoft/duroxide-pg#29:

  1. An orchestration acknowledge committed after another dispatcher had already taken the instance over.
  2. The messages that acknowledge consumed stayed queued, and the next owner processed them again.
  3. Replay does not de-duplicate queue messages or external events. So, reproduced end to end with duroxide 0.1.32, one enqueued event ran an activity twice.

The suite already checks some related cases:

  • a stale token that is rejected before the acknowledge starts (lock_expiration::test_lock_expires_after_timeout, which acknowledges an empty turn);
  • two acknowledges with the same token (atomicity::test_concurrent_ack_prevention).

It does not check that ownership holds until the acknowledge commits. It also has no positive test for orchestration lock renewal; the worker side has three.

All tests below use only the Provider trait and ProviderFactory::lock_timeout() (written T below). They should pass on the SQLite provider.

Contract changes (src/providers/mod.rs, docs/provider-implementation-guide.md)

  1. ack_orchestration_item: the lock token must still own the instance when the acknowledge commits, not only when it starts. If ownership was lost at any point, the acknowledge fails with a permanent error and commits nothing: no history, no new work, no deleted messages.
  2. renew_orchestration_item_lock: once the token no longer owns the instance (after an acknowledge, an abandon or a takeover), renewal fails and changes nothing. It must not lock released messages again or change the new owner's lease.
  3. Decide whether renewal may shorten a lease. The documented pattern is locked_until = now() + extend_for. Proposed: locked_until = max(locked_until, now() + extend_for). duroxide-pg does this since microsoft/duroxide-pg#44.

Tests to add

# Module Test Steps Must hold
1 lock_expiration test_ack_after_takeover_commits_nothing Fetch as A (lock T). Sleep past T. Fetch as B. Acknowledge as A with a non-empty history delta and one worker item A's acknowledge fails. read(instance) has none of A's events, and A's worker item was not enqueued. B's acknowledge succeeds, and no later fetch returns the message
2 lock_expiration test_orchestration_lock_renewal_extends_lock Fetch (lock T). Renew for 3T. Sleep 2T A second fetch returns nothing. An acknowledge with the original token succeeds
3 lock_expiration test_orchestration_renewal_after_takeover_fails Fetch as A. Wait for expiry. Fetch as B. Renew as A The renewal fails. A fetch right after it returns nothing. B's acknowledge succeeds and consumes the message once
4 lock_expiration test_orchestration_renewal_after_release_fails (a) Fetch, acknowledge, renew. (b) Fetch, abandon with no delay, renew Both renewals fail. In (b), a fetch right after the renewal returns the message: renewal did not lock it again
5 lock_expiration test_orchestration_renewal_never_shortens (after contract item 3) Fetch (lock T). Renew for 4T. Renew for T. Sleep 2T A second fetch returns nothing
6 instance_locking test_acknowledged_messages_never_redelivered_under_takeover_race N rounds (default 100). Each round: enqueue one QueueMessage. A fetches (lock T) and sleeps until about T, with jitter across rounds. Then A acknowledges a turn that consumes the message, while B polls fetch for up to 2T and acknowledges anything it gets with its own token. After every lock has expired, fetch once more Per round, at most one acknowledge that consumed the message succeeds. Once one succeeds, no later fetch returns the message

Notes:

  • Tests 1 to 4 document the contract. A provider that checks the token only at the start of an acknowledge still passes them, because the takeover happens before the call.
  • Test 5 fails on duroxide-pg main (3beba92), whose renewal can shorten a lease.
  • Test 6 is the one that targets microsoft/duroxide-pg#29. It is timing-based: measure its failure rate on duroxide-pg 3beba92 before relying on it. If the window is too narrow to hit reliably, keep it as a soak test with a configurable round count.

Acceptance

  • Tests 1 to 4, and test 5 once the contract is decided, pass on SQLite and on duroxide-pg with microsoft/duroxide-pg#44.
  • Test 5 fails on duroxide-pg 3beba92.
  • Test 6 passes on microsoft/duroxide-pg#44, and its failure rate on 3beba92 is recorded in the PR.
  • After the next duroxide release, duroxide-pg registers the new tests in tests/postgres_provider_test.rs.

Out of scope (these stay in duroxide-pg)

  • The clock-skew races, which need a faked caller clock that the trait cannot provide.
  • The PostgreSQL lock-order deadlock tests and the deadlock-victim retry.
  • The instance_locks(lock_token) index test.

These are in tests/deadlocks/ and tests/orchestration_*_tests.rs in microsoft/duroxide-pg#44. The TLA+ model in docs/tla/orchestration-leases there states the same properties formally.

Related

  • #55: tracking issue.
  • #58: the fallback abandon is dropped without .await.
  • #59: a turn keeps running after it loses its lock.
  • microsoft/duroxide-pg#29 and microsoft/duroxide-pg#44.
Lenguaje dominante
Rust
Estrellas
235
Forks
63
Merge medio
2 d 1 h
PR fusionados (30 d)
7

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/duroxide

Todos los issues de microsoft/duroxide

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.