[Bug]: Emulator retries invalid invocation outputs that the service fails immediately
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 72/100
Research direction
Start in aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation() and trace _validate_invocation_response_and_store() and _retry_invocation(). Reproduce the listed invalid-output cases and the PENDING case, then add regression coverage showing fail-fast behavior, the InvalidParameterValueException error shape, a three-attempt PENDING retry cap, and recovery after a transient invalid output.
Written by the indexing model from the issue text.
Description
Expected Behavior
The emulator should mirror the service's handling of invalid invocation outputs. Verified service behavior:
Most invalid outputs fail the execution immediately, with no retry. This applies to:
Status=FAILEDwith aResultpresentStatus=SUCCEEDEDwith anErrorpresent- a missing or unrecognized
Status - an unparseable/malformed invocation output payload
In each case the service fails the execution with an error object of the form:
{
"ErrorType": "InvalidParameterValueException",
"ErrorMessage": "<the specific validation message>"
}
There is exactly one retried case: Status=PENDING when the execution has no pending operations. The service treats this as a transient runtime-level error (it can arise from SDK race conditions that resolve on replay) and retries the invocation, up to 3 consecutive attempts, before failing the execution with the same InvalidParameterValueException-shaped error.
Actual Behavior
In aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation(), all validation failures from _validate_invocation_response_and_store() are routed through the same retry path:
except (InvalidParameterValueException, IllegalStateException) as e:
...
self._set_invocation_gate(execution_arn, InvocationState.PRE_INVOKE)
self._retry_invocation(execution, error_obj)
return
_retry_invocation re-invokes the handler up to MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5 times with a flat RETRY_BACKOFF_SECONDS = 5 backoff before failing.
Consequences:
- A deterministically invalid output (e.g.
SUCCEEDEDwith anError) is re-invoked 5 times before failing, where the service fails on the first response. For a deterministic handler this adds ~25 seconds and 4 pointless invocations, and every re-invocation replays the handler. - The one case the service does retry (
PENDINGwith no pending operations) gets 5 attempts instead of 3. - The terminal error surfaced to the customer is
ErrorObject.from_exception(e)rather than the service's{ErrorType: "InvalidParameterValueException", ErrorMessage: ...}shape.
Proposed Fix
In _finish_invocation, distinguish the two classes:
- Fail-fast validation errors (
FAILED+Result,SUCCEEDED+Error, missing/unknown status, malformed output): fail the execution immediately with anInvalidParameterValueException-typed error object carrying the validation message. PENDINGwith no pending operations: keep the retry path, with a consecutive-attempt cap of 3.
Regression tests:
- each fail-fast case produces a FAILED execution after a single invocation, with the
InvalidParameterValueExceptionerror shape; PENDINGwith no pending operations retries and fails after 3 consecutive attempts;- a transient case (invalid output once, then a valid
PENDING/SUCCEEDED) recovers without failing the execution.
Related
- Found while investigating https://github.com/aws/aws-durable-execution-sdk-python/issues/656 (FAILED without ErrorObject completed as SUCCEEDED by local emulator). This issue covers the adjacent divergence in the same code path: how invalid outputs are retried vs. failed.
SDK Version
aws-durable-execution-sdk-python-testing current main as of 2026-08-21.
Python Version
3.13
Is this a regression?
No known working version.
- Dominant language
- Python
- Stars
- 59
- Forks
- 32
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 76
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/aws-durable-execution-sdk-python
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aws/aws-durable-execution-sdk-python#977 ·
Maintainers usually reply within 1 day
-
[AgentCore session] Remove the unused session timeoutPossibly taken @srivilliamsai claimed this today. Openneeds-triage
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
aws/aws-durable-execution-sdk-python#970 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aws/aws-durable-execution-sdk-python#952 ·
Maintainers usually reply within 1 day
-
[AgentCore session] W4: Document sessions in the worker READMEPossibly taken @ParidelPooya claimed this 3 days ago. Openenhancement needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 76/100
aws/aws-durable-execution-sdk-python#950 ·
Maintainers usually reply within 1 day
-
[AgentCore session] W3: Add the session ID and job count to worker logsPossibly taken @ParidelPooya claimed this 3 days ago. Openenhancement needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
aws/aws-durable-execution-sdk-python#949 ·
Maintainers usually reply within 1 day
All issues in aws/aws-durable-execution-sdk-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
Unused loot bag group names left in dreamcraft en_US.langPossibly taken A pull request linked to this issue is open or already merged. OpenStatus: Triage Suggestion
Difficulty 1/5 Under an hour Newbie friendliness 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
Maintainers usually reply within 1 day
-
bug frontend sev-5
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
-
core good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
b4b code health next
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ESCOMP/CTSM#4330 · 1 comment ·
Maintainers usually reply within 1 day