[Epic]: Remaining Python OpenTelemetry work

Start with the listed child issues, especially #853–#863, and read PR #796 alongside conformance requirements in aws/aws-durable-execution-conformance-tests#131. Each child issue defines its own reproduction, design, acceptance criteria, and tests; pick one child issue to work on independently. The parent is complete when every child has a merged fix or documented disposition and the deferred conformance cases are qualified or their gaps explicitly recorded.

Parent tracker for the remaining Python OpenTelemetry fixes, core/testing prerequisites, and conformance qualification. Deliver the work through small, independently reviewable PRs; each issue owns its reproduction, design choices, acceptance criteria, and tests.

Status reviewed on 2026-11-02. Of the 13 native sub-issues, 5 are closed and 8 remain open. The original split from closed PRs #758 and #795 remains below; newer OTel follow-ups are listed separately by reference.

Latest progress

• Merged fixes: terminal status/event timing (#865 → #853), isolated terminal error snapshots (#866 → #852), empty-error preservation (#870 → #862), and callback update/input acknowledgment semantics (#898 → #860 and #861).
• #796 merged on November 2, delivering conformance cases 22–27, including the previously deferred cases 25–26. Its latest four-job cloud run passed 26 cases per job, with case 21 explicitly NOT_IMPLEMENTED. The additional case 17 callback-error regression is still outstanding.
• #894's partial callback-error type/message fix also merged in #901. Stacktrace emission remains separate work in #886.
• Replay-completion discovery (#854, open PR #869), export recovery (#855), bounded flushing (#856), the remaining core/testing items, and the newer follow-ups below remain open. Passing the current conformance catalog does not close those broader gaps.

OTel runtime correctness — original sub-issues

• #853 — Report terminal failures and keep exception events within span bounds — merged #865.
• #854 — Export unvisited replay completions with the correct parent context — open PR #869; broader skipped-parent/unvisited-completion coverage remains outstanding despite case 26 passing.
• #855 — Recover failed completion exports and isolate invocation finalization errors.
• #856 — Bound provider flush waiting without blocking unrelated providers.

Core instrumentation contract — original sub-issues

• #852 — Isolate terminal plugin error snapshots from checkpointed state — merged #866.
• #772 — Preserve handler errors in invocation-end hooks when returning PENDING.

Local-runner parity and installed artifacts — original sub-issues

• #858 — Prune completed-context descendants from replay state.
• #859 — Match ReplayChildren lifecycle and wire representation.
• #860 — Record callback terminal completions as updated operations — merged #898.
• #861 — Acknowledge invocation input without losing pending updates — merged #898.
• #862 — Preserve empty callback error history without changing runtime errors — merged #870.
• #863 — Validate the built testing-library wheel in OTel compatibility lanes.

#898 derives updated-operation IDs from the runner's existing touch sequences and the last successful invocation watermark. It also makes the duplicate terminal-STEP export reported in #892 reproducible locally; that SDK/OTel defect remains open. The earlier #867/#868 approaches are superseded by #898 and are not additional prerequisites.

Conformance and integration

• #874 — Complete replay and chained-execution conformance coverage — remains open for case 21, the additional case 17 regression, and completion of the remaining qualification work.

Coverage	Current state
Case 11 — chained-invoke replay	Delivered by merged #864; retained in the passing cloud matrix.
Cases 22–24 and 27 — user/callback context, invocation retry, suspended chained child	Delivered by merged #796 and passed in all four cloud jobs.
Cases 25–26 — caught callback-failure replay and external callback completion	Active in both cloud views and local tests; passed in all four cloud jobs. #865/#870 and #898 satisfy their merged prerequisites. They are no longer deferred pending #854.
Case 21 — completed-step replay	Local replay tests remain; both cloud views declare NOT_IMPLEMENTED because the shared history requirement assumes an event order the service can reverse.
Additional case 17 callback-error regression	Still to restore. The existing case 17 handler remains active. Source is preserved at 6484e577; correct detailed/metadata-only history expectations before restoring it.

Verified cloud evidence: run 41499388644 tested SDK revision de94078 against shared catalog 0eaf734. Invocation and Execution views each passed on Community/S3 and ADOT/X-Ray: 26 passed, 0 failed, 0 optional failed, 1 not implemented, 0 uncovered, 27 total per job. This qualifies the active catalog at those revisions, not case 21 or every open runtime issue. #796 also records 94 passing local conformance/workflow tests.

Remaining conformance work and constraints:

• Resolve the case 21 shared history-order requirement before re-enabling cloud coverage. The recorded failing run observed WaitSucceeded before InvocationCompleted on a successful execution. Preserve saved-step, result, and telemetry assertions when accommodating valid ordering; increasing the wait only reduces the race.
• Preserve case 26's observation that completion has reached the exporter before the target step runs, and its raw export-count assertions at later barriers. #796 records observations inside callbacks and asserts on the pytest thread so SDK retries cannot consume assertion failures.
• Detailed case 25 histories in run 40769339007, documented in #796, verified Error: {Payload: {}, Truncated: false} against Lambda. The metadata-only Error: {Truncated: true} shape remains unverified against deployed Lambda. Included diagnostic payloads are untruncated; metadata-only history cannot reconstruct omitted error details.
• Shared requirements/orchestration from aws/aws-durable-execution-conformance-tests#131 are merged. Coordinate remaining language-neutral changes in that repository. #796 pins the reusable workflow to 0eaf7341c0c7ff937e6a4bf6bda1682ff07e7502; the catalog defaults to shared main, with the manual conformance_test_ref override retained.

Newer OTel and instrumentation follow-ups

These are tracked here by reference and are not included in the 13 native sub-issue count. Broader core/plugin or cross-SDK changes retain the scope and compatibility requirements in their own issues.

Open issue	Remaining work
#892	Suppress reconstruction of already-terminal STEP spans on invocation retry in both views while preserving real attempts and non-STEP completion delivery. Best-effort suppression does not guarantee lossless or exactly-once export.
#893	Align Execution-view log correlation with Workflow/operation spans while preserving tracing scopes and Invocation-view behavior.
#886	Emit exception stacktraces only when supplied; do not manufacture a stack from a carrier exception.
#888	Record supplied terminal execution exceptions on Workflow and Invocation spans in both views.
#887	Honor the configured provider sampler and standard OTEL_TRACES_SAMPLER policies for durable spans.
#889	Preserve replay classification and resumed-operation notifications through virtual child contexts.
#890	Add virtual-parent instrumentation metadata and preserve logical virtual-context span hierarchy without changing checkpoint/history parentage.
#891	Preserve OTel registration metadata in test factory wrappers; open PR #895.
#882	Expose authoritative STOPPED/TIMED_OUT execution outcomes when a surviving invocation reaches finalization. This requires the shared core/plugin lifecycle contract, beyond terminal operation status mapping.
#883	Define supported telemetry coverage/recovery for hard Lambda invocation timeouts, where End hooks and queued exporter I/O may never run.
#884	Coordinate SHA-256 operation/trace/span ID derivation across SDKs and plugins, including persisted-operation identity and replay compatibility.

Related work with broader ownership

• #742 owns deployed Python LMI observability/X-Ray coverage alongside lifecycle tests. It already has Java #798 — Durable Functions on LMI as its parent; retain that hierarchy and track the Python OTel acceptance here by reference.
• #716 and #487 track cross-package release/publishing behavior, including the OTel package. Their broader release scope remains separate; link any concrete OTel release blocker discovered while delivering this work.
• #708 tracks log-visibility flakiness in the general plugin conformance suite. It is related test-infrastructure work, not evidence that an OTel runtime fix has passed or failed.

Working independently

Start each implementation from current main, use the issue's focused regression, and keep unrelated runtime/testing changes in their own PRs. Coordinate overlapping files without recreating the closed implementation stack. Test the combined fixes through the public SDK and real exporters after they land. A closed issue must link the merged fix or an explicit resolution; do not close this parent merely because a tracking PR exists or the currently enabled catalog passes.

Completion criteria

• All native sub-issues have merged fixes or an explicitly documented disposition; newer follow-ups have a clear resolution or ownership/scope disposition.
• Replay preserves saved successes/failures, operation identity, and completed-work skipping; completion telemetry has the correct parent, status, event bounds, and export phase in both views, without reconstructing already-terminal STEP work.
• Export-failure cleanup and provider-flush behavior satisfy the bounds and ownership rules in their issues without claiming exactly-once delivery.
• Runner parity is checked across memory/filesystem stores, replay/state reads, callback updates, and history serialization; installed-wheel lanes exercise the built testing library.
• Cases 22–27, including the previously deferred cases 25–26, are delivered and qualified in the four-job cloud matrix above.
• Cloud case 21 and the additional case 17 callback-error regression are restored and qualified, or explicitly dispositioned with evidence; missing coverage is never counted as passing.
• Python LMI OTel acceptance is linked from #742, with deployed evidence appropriate to Python's process model.
• Documentation and validation results reflect the final behavior, and external assumptions have evidence or are marked unverified. The metadata-only callback-error service shape remains unverified; detailed empty-error history now has cloud evidence.