A failed commit becomes a permanent OrchestrationFailed without parent notification; the fallback abandon never runs
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- rust
- Área
- distributed-systems
Línea de trabajo
Start with process_orchestration_item and ack_orchestration_with_changes in src/runtime/dispatchers/orchestration.rs, then read the related design in #55. Trace the retry and error branches, including both abandon_orchestration_item calls; done means the chosen failure behavior executes asynchronously and does not leave parent notification or activity cancellation missing.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
When ack_orchestration_item fails, the runtime commits an OrchestrationFailed event for the instance. It does this for every error that the provider calls non-retryable, and for a retryable error that lasts longer than about 310 ms. So a database problem that goes away can end an orchestration for good.
The same code path has two more problems:
- The failure is committed without messages. A parent orchestration gets no
SubOrchFailedand waits forever. - The fallback call to
abandon_orchestration_itemnever runs.
Where
process_orchestration_item, the ack and its error branch: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L949-L1017ack_orchestration_with_changes, the retry loop: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L1161-L1210
Details
- Retry budget. A retryable error is retried 5 times, with waits of 10, 20, 40, 80 and 160 ms. Then the error is returned (L1191-L1206). A non-retryable error is returned at once (L1183-L1188).
- Failure commit. The caller builds
OrchestrationFailedfrom the history it fetched and commits it with the same lock token (L975-L1001). If the database is healthy again at that moment, the commit works and the instance isFailed. - No messages. That commit passes empty
worker_items,orchestrator_itemsandcancelled_activities(L990-L999). If the instance is a sub-orchestration, its parent is not told. In-flight activities are not cancelled. - The abandon never runs. Both fallback paths call
drop(self.history_store.abandon_orchestration_item(...))(L1009-L1013 and L1200-L1204).abandon_orchestration_itemis anasync fn. A future that is dropped without.awaitdoes nothing. The lock is only released when it times out.
Which errors are non-retryable depends on the provider. duroxide-pg classifies almost every SQLSTATE as permanent, including a statement timeout: microsoft/duroxide-pg#30.
How this was checked
Found by reading the code. Not reproduced.
Suggested fix
- Do not fail the instance for an infrastructure error on commit. Abandon the item with a backoff, and let
max_attemptsend a message that can never commit. - If the failure commit stays, queue
SubOrchFailedfor the parent and cancel the in-flight activities, as the normal failure path does. .awaitthe twoabandon_orchestration_itemcalls.
Tracked in #55.
- Lenguaje dominante
- Rust
- Estrellas
- 221
- Forks
- 61
- Merge medio
- 3 d 5 h
- PR fusionados (30 d)
- 1
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de microsoft/duroxide
-
One failed session lock renewal loses the session: no retry, no log, no signal to running activitiesAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
-
bug
Dificultad 3/5 1-2 días Aptitud para principiantes 72/100
-
bug
Dificultad 5/5 Más de una semana Aptitud para principiantes 38/100
-
prune_kv_values_updated_before emits actions in HashMap order; replay fails with `kv clear mismatch`Posiblemente ocupada @akhil9tiet la tomó hace 5 días. Abiertobug
Dificultad 3/5 1-2 días Aptitud para principiantes 25/100
-
A failed activity lock renewal can lose the activity result; the orchestration waits foreverAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
Todos los issues de microsoft/duroxide
Issues similares
-
test(executor_l0): assert execute() TaskOutcome, not only bus events / 断言 execute() 返回的 TaskOutcomeAbiertotype:debt
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
skaiy/wild_agentos#425 ·
Los mantenedores suelen responder en 1 día
-
Default-import note suggests `import * as process` for velt:process, which does not name the builtinAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
bug ticket
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
cratestack/cratestack#1154 ·
Los mantenedores suelen responder en 1 día
-
status:needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
agentic-os-org/ANOLISA#6742 · 1 comentario ·
Los mantenedores suelen responder en 1 día