Orchestration completion and failure are logged and counted before the commit
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- distributed-systems
Research direction
Start in src/runtime/dispatchers/orchestration.rs, reading the log and metric code around lines 741-893, the first-turn metrics around 670-681, and the commit around 949-961. Trace ack_orchestration_item and verify that terminal logs and metrics occur only after a successful commit, including the lost-lock case described in the issue.
Written by the indexing model from the issue text.
Description
Summary
process_orchestration_item writes the "Orchestration completed" and "Orchestration failed" log lines, and records the completion and failure metrics, before it calls ack_orchestration_item. When the commit is rejected, the logs and the metrics describe something that did not happen.
Where
- Logs and metrics: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L741-L893
- The commit comes after them: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L949-L961
- The same holds for the start metric and the active gauge on a first turn: https://github.com/microsoft/duroxide/blob/6a458861763a7aa5b78a7c1c97691a6f00489a8b/src/runtime/dispatchers/orchestration.rs#L670-L681
Effects
- A turn that lost its lock still logs
Orchestration failedat ERROR, orOrchestration completed. Its commit is then rejected withInvalid lock token. An operator sees an error for an instance that is healthy. record_orchestration_completion_with_labels,record_orchestration_failure_with_labelsandrecord_continue_as_newcount a turn that was not committed. The turn that commits later is counted again.decrement_active_orchestrationsruns twice for one instance, so the active gauge drifts down.
How this was checked
Read in the code. A local reproduction of a lost lock showed the log pattern: the turn logged Orchestration failed at ERROR, and the next line was ack_orchestration_item failed with non-retryable error with Invalid lock token.
Suggested fix
Write the terminal logs and record the metrics after a successful commit. PR #49 does this for the poison path. This is the same change for the normal path.
Tracked in #55.
- Dominant language
- Rust
- Stars
- 221
- Forks
- 61
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/duroxide
-
One failed session lock renewal loses the session: no retry, no log, no signal to running activitiesOpenbug
Difficulty 4/5 3-5 days Newbie friendliness 30/100
-
bug
Difficulty 5/5 Over a week Newbie friendliness 38/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
prune_kv_values_updated_before emits actions in HashMap order; replay fails with `kv clear mismatch`Possibly taken @akhil9tiet claimed this 2 days ago. Openbug
Difficulty 3/5 1-2 days Newbie friendliness 25/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in microsoft/duroxide
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 4 days
-
`future_into_py` loses the original panic messagePossibly taken @Danipulok claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
PyO3/pyo3-async-runtimes#91 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 2 days