[Bug]: Emulator retries invalid invocation outputs that the service fails immediately

Open
#670 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation() and trace _validate_invocation_response_and_store() and _retry_invocation(). Reproduce the listed invalid-output cases and the PENDING case, then add regression coverage showing fail-fast behavior, the InvalidParameterValueException error shape, a three-attempt PENDING retry cap, and recovery after a transient invalid output.

Written by the indexing model from the issue text.

Description

bug parity pkg:testing
Expected Behavior

The emulator should mirror the service's handling of invalid invocation outputs. Verified service behavior:

Most invalid outputs fail the execution immediately, with no retry. This applies to:

  • Status=FAILED with a Result present
  • Status=SUCCEEDED with an Error present
  • a missing or unrecognized Status
  • an unparseable/malformed invocation output payload

In each case the service fails the execution with an error object of the form:

{
  "ErrorType": "InvalidParameterValueException",
  "ErrorMessage": "<the specific validation message>"
}

There is exactly one retried case: Status=PENDING when the execution has no pending operations. The service treats this as a transient runtime-level error (it can arise from SDK race conditions that resolve on replay) and retries the invocation, up to 3 consecutive attempts, before failing the execution with the same InvalidParameterValueException-shaped error.

Actual Behavior

In aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation(), all validation failures from _validate_invocation_response_and_store() are routed through the same retry path:

except (InvalidParameterValueException, IllegalStateException) as e:
    ...
    self._set_invocation_gate(execution_arn, InvocationState.PRE_INVOKE)
    self._retry_invocation(execution, error_obj)
    return

_retry_invocation re-invokes the handler up to MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5 times with a flat RETRY_BACKOFF_SECONDS = 5 backoff before failing.

Consequences:

  • A deterministically invalid output (e.g. SUCCEEDED with an Error) is re-invoked 5 times before failing, where the service fails on the first response. For a deterministic handler this adds ~25 seconds and 4 pointless invocations, and every re-invocation replays the handler.
  • The one case the service does retry (PENDING with no pending operations) gets 5 attempts instead of 3.
  • The terminal error surfaced to the customer is ErrorObject.from_exception(e) rather than the service's {ErrorType: "InvalidParameterValueException", ErrorMessage: ...} shape.
Proposed Fix

In _finish_invocation, distinguish the two classes:

  • Fail-fast validation errors (FAILED+Result, SUCCEEDED+Error, missing/unknown status, malformed output): fail the execution immediately with an InvalidParameterValueException-typed error object carrying the validation message.
  • PENDING with no pending operations: keep the retry path, with a consecutive-attempt cap of 3.

Regression tests:

  • each fail-fast case produces a FAILED execution after a single invocation, with the InvalidParameterValueException error shape;
  • PENDING with no pending operations retries and fails after 3 consecutive attempts;
  • a transient case (invalid output once, then a valid PENDING/SUCCEEDED) recovers without failing the execution.
Related
SDK Version

aws-durable-execution-sdk-python-testing current main as of 2026-08-21.

Python Version

3.13

Is this a regression?

No known working version.

Dominant language
Python
Stars
59
Forks
32
Avg merge
1d 5h
Merged PRs (30d)
76

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/aws-durable-execution-sdk-python

All issues in aws/aws-durable-execution-sdk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.