TEST Cover cancelled scenario readback cleanup
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 76/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- backend, testing-qa
Research direction
Start in tests/unit/backend/test_scenario_run_service.py and read the existing cancellation, terminal-progress, and active-task cleanup tests around ScenarioRunService. Use an asyncio.Event to control the blocked run_async() task, then exercise cancel_run_async() and get_run(); done means the persisted state reads as CANCELLED and the stale _active_tasks entry is cleaned up, with the focused module and listed validation checks passing.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
ScenarioRunService has deterministic coverage for cancellation, terminal progress, and deferred active-task cleanup after success or failure, but it does not directly cover the cancelled readback boundary:
- a scenario run is blocked inside
run_async(); cancel_run_async()cancels and awaits that task;- persistence reports the run as
ScenarioRunState.CANCELLED; get_run()reads the terminal database state;- the completed/cancelled entry is removed from
_active_tasks.
No production defect has been reproduced. The missing regression matters because a stale active-task entry could make the in-memory lifecycle state disagree with the database, retain task/scenario objects longer than necessary, or affect subsequent status reads.
Describe the solution you'd like
Add a deterministic unit test in tests/unit/backend/test_scenario_run_service.py that:
- starts or installs a scenario task whose
run_async()is blocked on anasyncio.Event; - waits until execution is definitely blocked before cancelling;
- calls and awaits
cancel_run_async(); - configures the persisted scenario header/result to report
CANCELLED; - calls
get_run(); - asserts the returned state is the terminal persisted
CANCELLEDstate; - verifies the stale active-task entry is cleaned up.
Prefer asserting observable behavior where possible. If direct inspection of _active_tasks is necessary to prove cleanup, keep that assertion narrowly scoped and document the intended internal invariant through the test name and setup rather than changing the public API solely for testing.
The test should control task scheduling with events rather than sleeps, verify cancellation is fully awaited, and ensure cleanup happens exactly once.
Describe alternatives you've considered, if relevant
Existing tests separately cover cancelled progress deltas and active-task cleanup after successful or failed runs, but neither proves the combined cancelled-readback path. A broad integration test would be slower and less deterministic than a focused service unit test.
Changing production code without first reproducing a failure is not proposed. If the test exposes a mismatch between persisted terminal state and in-memory cleanup, the smallest behavior-preserving fix should be included with the regression.
Additional context
This boundary was identified during the September 12, 2026 deterministic resilience audit with medium-high confidence. The audit found no exact existing issue or test covering it.
Suggested validation:
- the new focused
ScenarioRunServicetest; - the complete backend scenario-run service test module;
- cancellation and progress route tests if production code changes;
- Ruff, typing, and
git diff --check.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 155
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Bug: triage help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Bug: triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches Open
Difficulty 3/5 1-2 days Newbie friendliness 78/100
-
Bug: triage help wanted
Difficulty 3/5 1-2 days Newbie friendliness 75/100
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100