Orchestration completion and failure are logged and counted before the commit
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 72/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- rust
- Área
- distributed-systems
Línea de trabajo
Start in src/runtime/dispatchers/orchestration.rs, reading the log and metric code around lines 741-893, the first-turn metrics around 670-681, and the commit around 949-961. Trace ack_orchestration_item and verify that terminal logs and metrics occur only after a successful commit, including the lost-lock case described in the issue.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
process_orchestration_item writes the "Orchestration completed" and "Orchestration failed" log lines, and records the completion and failure metrics, before it calls ack_orchestration_item. When the commit is rejected, the logs and the metrics describe something that did not happen.
Where
- Logs and metrics: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L741-L893
- The commit comes after them: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L949-L961
- The same holds for the start metric and the active gauge on a first turn: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L670-L681
Effects
- A turn that lost its lock still logs
Orchestration failedat ERROR, orOrchestration completed. Its commit is then rejected withInvalid lock token. An operator sees an error for an instance that is healthy. record_orchestration_completion_with_labels,record_orchestration_failure_with_labelsandrecord_continue_as_newcount a turn that was not committed. The turn that commits later is counted again.decrement_active_orchestrationsruns twice for one instance, so the active gauge drifts down.
How this was checked
Read in the code. A local reproduction of a lost lock showed the log pattern: the turn logged Orchestration failed at ERROR, and the next line was ack_orchestration_item failed with non-retryable error with Invalid lock token.
Suggested fix
Write the terminal logs and record the metrics after a successful commit. PR #49 does this for the poison path. This is the same change for the normal path.
Tracked in #55.
- Lenguaje dominante
- Rust
- Estrellas
- 221
- Forks
- 61
- Merge medio
- 3 d 5 h
- PR fusionados (30 d)
- 1
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de microsoft/duroxide
-
One failed session lock renewal loses the session: no retry, no log, no signal to running activitiesAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
-
bug
Dificultad 5/5 Más de una semana Aptitud para principiantes 38/100
-
bug
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
-
prune_kv_values_updated_before emits actions in HashMap order; replay fails with `kv clear mismatch`Posiblemente ocupada @akhil9tiet la tomó hace 4 días. Abiertobug
Dificultad 3/5 1-2 días Aptitud para principiantes 25/100
-
A failed activity lock renewal can lose the activity result; the orchestration waits foreverAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
Todos los issues de microsoft/duroxide
Issues similares
-
mxl-compile: пример заполнения ячеек отклоняется UnicaPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto
Dificultad 1/5 1-3 horas Aptitud para principiantes 91/100
IngvarConsulting/unica#1301 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
curator: add tutros/sbxmAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 75/100
ajeetraina/awesome-docker-sbx#220 ·
-
`helios / deploy`: switch zone wait in `deploy.sh` has almost no headroom over healthy startup timesPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. AbiertoTest Flake
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
oxidecomputer/omicron#11453 ·
Los mantenedores suelen responder en 1 día