[Bug]: Resumed map or parallel can omit recorded terminal item outcomes
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- backend, distributed-systems
Research direction
Start with concurrency/executor.py, then trace how operation/map.py and operation/parallel.py call execute() for a STARTED parent. Reproduce the recorded SUCCEEDED, SUCCEEDED, FAILED child state with the deterministic event ordering described in the issue. Done means both APIs retain every terminal child outcome before applying completion policies, and the listed regression cases pass without changing checkpoint formats.
Written by the indexing model from the issue text.
Description
Reference implementation
Merged JavaScript fix: aws/aws-durable-execution-sdk-js#1091 — keep recorded item outcomes when a batch resumes
The JavaScript PR fixes the equivalent defect and includes reconciliation tests for recorded successes, recorded failures, bounded concurrency, unfinished items, and custom completion predicates. Its required behavior and regression cases are the parity reference for this issue. The Python implementation can use a different internal design, but it should produce the same observable result for recorded terminal items.
Expected Behavior
A map or parallel operation can resume while its parent CONTEXT operation is still STARTED. Some nested item or branch contexts can already have terminal SUCCEEDED or FAILED records.
The resumed batch must incorporate every existing terminal child-context outcome before it makes a new completion decision.
For this state:
| Operation | Recorded status |
|---|---|
| Batch parent | STARTED |
| Item 0 | SUCCEEDED |
| Item 1 | SUCCEEDED |
| Item 2 | FAILED |
| Item 3 | STARTED |
with:
MapConfig(
max_concurrency=2,
completion_config=CompletionConfig(min_successful=2),
)
the resumed BatchResult must satisfy:
[item.status for item in result.all] == [
BatchItemStatus.SUCCEEDED,
BatchItemStatus.SUCCEEDED,
BatchItemStatus.FAILED,
]
result.success_count == 2
result.failure_count == 1
result.has_failure is True
result.status is BatchItemStatus.FAILED
result.completion_reason is CompletionReason.MIN_SUCCESSFUL_REACHED
result.throw_if_error() must raise the recorded item failure.
Item 3 does not need to remain in the result. A previously STARTED item is not a terminal outcome.
Actual Behavior
When the batch parent is STARTED, map_handler() and parallel_handler() call ConcurrentExecutor.execute().
execute() initializes every branch as pending. It does not first identify child contexts that already have terminal records.
The recorded child contexts replay on worker threads. Their events reach the coordinator in thread-completion order. That order can differ from their original completion order and their item-index order.
In the state above:
- Items 0 and 1 replay their recorded successes.
- The first success frees a concurrency slot, so item 2 is submitted.
- The second success reaches
min_successful=2. - The coordinator completes the batch before item 2's worker reports its already-recorded failure.
- The final nonblocking event drain does not wait for that event.
The resulting value can be:
[item.status for item in result.all] == [
BatchItemStatus.SUCCEEDED,
BatchItemStatus.SUCCEEDED,
BatchItemStatus.STARTED,
]
result.failure_count == 0
result.has_failure is False
result.status is BatchItemStatus.SUCCEEDED
The execution history still contains item 2's FAILED child-context record.
The resumed parent then checkpoints this incomplete BatchResult. Subsequent replay preserves the incorrect result.
Steps to Reproduce
-
Configure a four-item nested map with:
MapConfig( max_concurrency=2, completion_config=CompletionConfig(min_successful=2), nesting_type=NestingType.NESTED, ) -
Produce this first-invocation order:
- Items 0 and 1 start.
- Item 1 succeeds, which starts item 2.
- Item 2 fails and records its terminal child-context failure.
- Item 3 starts.
- Item 0 succeeds, which reaches
min_successful=2. - End the invocation before the batch parent records its terminal checkpoint.
-
Resume with the batch parent still
STARTEDand child records:item 0: SUCCEEDED item 1: SUCCEEDED item 2: FAILED item 3: STARTED -
In a deterministic unit test:
- Allow item 0's cached success event to reach the coordinator.
- Confirm that the coordinator submits item 2.
- Hold item 2 after it observes the recorded failure but before its worker posts the
FAILEDevent. - Allow item 1's cached success event to reach the coordinator.
- Observe that the coordinator returns after reaching two successes.
-
Assert that item 2 is reported as
STARTED,failure_countis zero, andhas_failureis false despite its recordedFAILEDcontext.
The same coordinator code is used by parallel().
SDK Version
Reproduced in:
aws-durable-execution-sdk-python2.1.0aws-durable-execution-sdk-python3.0.0 on currentmain
The relevant executor, map handler, and parallel handler are unchanged between these versions.
Python Version
3.14
Is this a regression?
No.
Additional Context
Relevant Python code:
concurrency/executor.py- Evaluates completion after each worker event.
- Stops once the current counts meet the completion policy.
- Performs one nonblocking event drain after that decision.
- Does not identify terminal child records before scheduling.
operation/map.py- Calls
replay()only for a terminal parent. - Calls
execute()when the parent remainsSTARTED.
- Calls
operation/parallel.py- Uses the same distinction.
concurrency/models.py- Calculates
has_failure, status, and errors only from items included in the returnedBatchResult.
- Calculates
Success criteria
-
Recorded outcomes are retained
- Every existing terminal
SUCCEEDEDorFAILEDnested child-context record appears with that terminal status in the resumedBatchResult. - A terminal record cannot become
STARTEDor disappear because of worker-event ordering.
- Every existing terminal
-
Completion waits for reconciliation
- The resumed batch does not resolve until all child contexts that were terminal at invocation start have been incorporated.
- The order in which their worker events arrive does not change which recorded outcomes appear.
-
Completion policies use the reconciled state
min_successful, fail-fast, failure-count tolerance, and failure-percentage tolerance are evaluated against all recorded terminal outcomes before later items are admitted.- A supported monotonic
should_completepredicate cannot cause a recorded terminal outcome to be omitted.
-
No unnecessary new work starts
- If the reconciled terminal records already satisfy a completion rule, no item beyond the recorded range starts.
- Existing operation-ID ordering and
max_concurrencyguarantees remain intact.
-
Existing replay behavior remains valid
- A parent that is already terminal continues replaying its serialized
BatchResultorCompletionRecord. - Existing normal-payload and
ReplayChildrenhistories require no migration. - Existing in-flight
STARTEDexecutions remain readable without a checkpoint-format change.
- A parent that is already terminal continues replaying its serialized
-
Tests cover both APIs
- Add the exact
SUCCEEDED,SUCCEEDED,FAILEDreproduction formap(). - Add equivalent coverage for
parallel(). - Cover fail-fast and failure-tolerance decisions.
- Cover delayed deserialization or delayed worker-event delivery.
- Assert that
has_failure,failure_count,status, andthrow_if_error()reflect the recorded failure. - Assert that terminal branch bodies follow their existing replay behavior.
- Assert that later item bodies do not run once the reconciled state has completed the batch.
- Add the exact
-
Documentation distinguishes two cases
- A
STARTEDparent must reconcile terminal child records before completing. - After the parent itself has checkpointed its
BatchResult, that result remains authoritative on replay. A child can still appear terminal in history while the recorded batch item remainsSTARTED, as documented for JavaScript by aws/aws-durable-execution-sdk-js#1092.
- A
Out of scope
- Changing
NestingType.FLAT, which has no per-item child-context record to reconcile. - Changing the accepted result/history mismatch after the batch parent has recorded its result.
- Rejecting new operations under orphaned branches, tracked by #641.
- Invocation deadlines and unbounded branch cleanup, tracked by #741.
- Changing any public method signature or checkpoint payload format.
- Dominant language
- Python
- Stars
- 59
- Forks
- 32
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 76
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/aws-durable-execution-sdk-python
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aws/aws-durable-execution-sdk-python#977 ·
Maintainers usually reply within 1 day
-
[AgentCore session] Remove the unused session timeoutPossibly taken @srivilliamsai claimed this today. Openneeds-triage
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
aws/aws-durable-execution-sdk-python#970 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aws/aws-durable-execution-sdk-python#952 ·
Maintainers usually reply within 1 day
-
[AgentCore session] W4: Document sessions in the worker READMEPossibly taken @ParidelPooya claimed this 3 days ago. Openenhancement needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 76/100
aws/aws-durable-execution-sdk-python#950 ·
Maintainers usually reply within 1 day
-
[AgentCore session] W3: Add the session ID and job count to worker logsPossibly taken @ParidelPooya claimed this 3 days ago. Openenhancement needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
aws/aws-durable-execution-sdk-python#949 ·
Maintainers usually reply within 1 day
All issues in aws/aws-durable-execution-sdk-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
Unused loot bag group names left in dreamcraft en_US.langPossibly taken A pull request linked to this issue is open or already merged. OpenStatus: Triage Suggestion
Difficulty 1/5 Under an hour Newbie friendliness 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
Maintainers usually reply within 1 day
-
bug frontend sev-5
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
-
core good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
b4b code health next
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ESCOMP/CTSM#4330 · 1 comment ·
Maintainers usually reply within 1 day