[Feature]: Define plugin telemetry coverage and recovery for hard Lambda invocation timeouts

Aberta
  • Algo errado?
#883 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
30/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
aws, python
Domínio
observability-sre

Direção de pesquisa

Start with packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/plugin.py and the telemetry exporter’s plugin.py, especially invocation-end emission and drain. Then run the specified cloud qualification across emit modes, resume, and terminal-without-resume cases, correlating platform logs, execution history, and exported records. Done means documenting supported loss and recovery guarantees, with focused tests that preserve retry semantics and do not claim post-kill callbacks.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement needs-decision pkg:plugins pkg:sdk
Problem

When Lambda terminates an invocation during a long-running step, that process cannot reliably execute its user-function/operation/invocation End hooks or finish queued exporter I/O. This leaves a gap in the telemetry for the interrupted invocation.

This affects the shared plugin lifecycle, including Durable Execution Telemetry's final record/drain/flush and OTel's open spans. It needs an explicit supported coverage and recovery contract. It is a platform/lifecycle limitation, not evidence that an in-process hook can execute after the runtime has already been killed.

Observable Impact

For the Durable Execution Telemetry plugin:

  • The interrupted invocation may never reach the End hook that emits its closing record and drains/flushes exporters.
  • In on-change mode, previously delivered RUNNING snapshots can remain available, but later queued or in-memory data is not guaranteed to arrive.
  • In on-complete/on-failure mode, the killed invocation cannot produce the corresponding final record through its normal End path.
  • If the durable execution resumes, a later invocation can export a new snapshot of persisted state. That does not recover all invocation-local timing or uncheckpointed work from the terminated process.
  • If the service later terminates the execution without another handler invocation, an in-process plugin alone may never emit the execution's final outcome.

For OTel, attempt/operation/invocation spans that remain open at the hard timeout lack normal End callbacks. Already-ended spans may have been exported before the timeout; this issue does not assume all spans are buffered until invocation End.

Under at-least-once-per-retry semantics, interruption can re-run the same durable step attempt. Repeated attempt number 1 is not itself a numbering defect when no failure was checkpointed. Do not fabricate extra step failures or retries merely to make the telemetry look complete.

Reproduction and Qualification
  1. Deploy a durable handler with a named step whose body remains active beyond the Lambda function timeout; for example, a 20-second test delay inside the step with a 10-second function timeout. Configure the durable execution timeout separately.
  2. Enable sampling and a telemetry exporter destination. Inspect on-change, on-complete and on-failure modes separately where supported.
  3. Correlate platform timeout logs, durable execution history and exported records by execution ARN. Inspect records delivered before the kill and the absence of normal finalization from the killed invocation.
  4. Exercise a timeout followed by a successful resumed invocation using a test-controlled external gate inside the step; use stable, checkpointed orchestration. Verify that completed operations are replayed and not executed again.
  5. Separately exercise exhaustion/terminal execution without another handler invocation. Identify which authoritative platform/service evidence remains available.
  6. Repeat OTel coverage with each view separately if qualifying trace visibility alongside telemetry records.

These steps describe the required cloud qualification; no new Lambda run was performed for this issue. Existing bug-bash reports describe missing SDK telemetry after hard timeouts, but exact deployed artifact versions and raw exporter records have not been independently re-read here.

Proposed Scope and Acceptance Criteria
  • Document which lifecycle hooks and exporter guarantees require orderly invocation finalization, and what a hard runtime timeout can lose.
  • Explain the telemetry emit-mode behavior above and how a later invocation's persisted-state snapshot differs from telemetry of the interrupted work.
  • Define the supported diagnostic/recovery path for invocations or executions with no subsequent handler run: platform logs, service history and/or an out-of-process collection path, with clear ownership.
  • Decide whether any bounded, best-effort pre-deadline emission is supported. If selected, state its limits and test it without claiming a guaranteed post-kill callback or exactly-once delivery. If unsupported, make the limitation explicit.
  • Add focused process/cloud qualification that distinguishes abrupt termination from a caught exception or ordinary PENDING suspension. Verify evidence before and after one resume, completed-work replay, and the terminal-without-resume case.
  • Preserve business outcomes, at-least-once retry semantics and recorded attempt counts. Do not manufacture durable operations, failures, start/end times or successful exporter acknowledgements.
  • Keep any OTel-specific visualization work separate from the common SDK/plugin delivery contract.
Source and Affected Packages

Reviewed Python source at 44b9f4d1; this identifies inspected source, not a newly verified deployment.

The telemetry plugin consumes operation snapshots rather than accumulating operation-End events. This finding concerns losing the finalization/export opportunity when the runtime dies, not duplicate completion notification counting.

Related Work
  • #882 concerns stop or durable execution timeout while the current invocation survives and can potentially obtain authoritative terminal status. Here the Lambda runtime is terminated, so that in-process lookup opportunity is absent.
  • Invocation-deadline/worker-cleanup improvements can reduce some exposure but cannot guarantee that arbitrary blocked work or exporter I/O completes before a hard platform kill.
  • This is the Python SDK tracking item for the cross-SDK hard-invocation-timeout telemetry finding (F19).

Cross-SDK counterpart: https://github.com/aws/aws-durable-execution-sdk-js/issues/1128

Linguagem predominante
Python
Estrelas
58
Forks
32
Merge médio
1d 6h
PRs com merge (30d)
80

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de aws/aws-durable-execution-sdk-python

Todas as issues de aws/aws-durable-execution-sdk-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.