Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Feature]: Define plugin telemetry coverage and recovery for hard Lambda invocation timeouts

Abierto
  • ¿Algo no está bien?
#850 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
aws, java
Área
backend, observability

Línea de trabajo

Start with DurableExecutionPlugin.java and TelemetryPlugin.java at the reviewed commit; trace the invocation-end emission and exporter drain path. Then read the cross-SDK counterpart and issue #848 to understand the boundary of this scope. Define the supported loss and recovery contract, and add focused process or cloud qualification for timeout, resume, and terminal-without-resume cases while preserving retry and attempt semantics.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement needs-decision pkg:plugins pkg:sdk
Problem

When Lambda terminates an invocation during a long-running step, that process cannot reliably execute its user-function/operation/invocation End hooks or finish queued exporter I/O. This leaves a gap in the telemetry for the interrupted invocation.

This affects the shared plugin lifecycle, including Durable Execution Telemetry's final record/drain/flush and OTel's open spans. It needs an explicit supported coverage and recovery contract. It is a platform/lifecycle limitation, not evidence that an in-process hook can execute after the runtime has already been killed.

Observable Impact

For the Durable Execution Telemetry plugin:

  • The interrupted invocation may never reach the End hook that emits its closing record and drains/flushes exporters.
  • In on-change mode, previously delivered RUNNING snapshots can remain available, but later queued or in-memory data is not guaranteed to arrive.
  • In on-complete/on-failure mode, the killed invocation cannot produce the corresponding final record through its normal End path.
  • If the durable execution resumes, a later invocation can export a new snapshot of persisted state. That does not recover all invocation-local timing or uncheckpointed work from the terminated process.
  • If the service later terminates the execution without another handler invocation, an in-process plugin alone may never emit the execution's final outcome.

For OTel, attempt/operation/invocation spans that remain open at the hard timeout lack normal End callbacks. Already-ended spans may have been exported before the timeout; this issue does not assume all spans are buffered until invocation End.

Under at-least-once-per-retry semantics, interruption can re-run the same durable step attempt. Repeated attempt number 1 is not itself a numbering defect when no failure was checkpointed. Do not fabricate extra step failures or retries merely to make the telemetry look complete.

Reproduction and Qualification
  1. Deploy a durable handler with a named step whose body remains active beyond the Lambda function timeout; for example, a 20-second test delay inside the step with a 10-second function timeout. Configure the durable execution timeout separately.
  2. Enable sampling and a telemetry exporter destination. Inspect on-change, on-complete and on-failure modes separately where supported.
  3. Correlate platform timeout logs, durable execution history and exported records by execution ARN. Inspect records delivered before the kill and the absence of normal finalization from the killed invocation.
  4. Exercise a timeout followed by a successful resumed invocation using a test-controlled external gate inside the step; use stable, checkpointed orchestration. Verify that completed operations are replayed and not executed again.
  5. Separately exercise exhaustion/terminal execution without another handler invocation. Identify which authoritative platform/service evidence remains available.
  6. Repeat OTel coverage with each view separately if qualifying trace visibility alongside telemetry records.

These steps describe the required cloud qualification; no new Lambda run was performed for this issue. Existing bug-bash reports describe missing SDK telemetry after hard timeouts, but exact deployed artifact versions and raw exporter records have not been independently re-read here.

Proposed Scope and Acceptance Criteria
  • Document which lifecycle hooks and exporter guarantees require orderly invocation finalization, and what a hard runtime timeout can lose.
  • Explain the telemetry emit-mode behavior above and how a later invocation's persisted-state snapshot differs from telemetry of the interrupted work.
  • Define the supported diagnostic/recovery path for invocations or executions with no subsequent handler run: platform logs, service history and/or an out-of-process collection path, with clear ownership.
  • Decide whether any bounded, best-effort pre-deadline emission is supported. If selected, state its limits and test it without claiming a guaranteed post-kill callback or exactly-once delivery. If unsupported, make the limitation explicit.
  • Add focused process/cloud qualification that distinguishes abrupt termination from a caught exception or ordinary PENDING suspension. Verify evidence before and after one resume, completed-work replay, and the terminal-without-resume case.
  • Preserve business outcomes, at-least-once retry semantics and recorded attempt counts. Do not manufacture durable operations, failures, start/end times or successful exporter acknowledgements.
  • Keep any OTel-specific visualization work separate from the common SDK/plugin delivery contract.
Source and Affected Packages

Reviewed Java source at 4b1aa82b; this identifies inspected source, not a newly verified deployment.

The telemetry plugin consumes operation snapshots rather than accumulating operation-End events. This finding concerns losing the finalization/export opportunity when the runtime dies, not duplicate completion notification counting.

Related Work
  • #848 concerns stop or durable execution timeout while the current invocation survives and can potentially obtain authoritative terminal status. Here the Lambda runtime is terminated, so that in-process lookup opportunity is absent.
  • Invocation-deadline/worker-cleanup improvements can reduce some exposure but cannot guarantee that arbitrary blocked work or exporter I/O completes before a hard platform kill.
  • This is the Java SDK tracking item for the cross-SDK hard-invocation-timeout telemetry finding (F19).

Cross-SDK counterpart: https://github.com/aws/aws-durable-execution-sdk-js/issues/1128

Lenguaje dominante
Java
Estrellas
28
Forks
16
Merge medio
2 d 59 min
PR fusionados (30 d)
62

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de aws/aws-durable-execution-sdk-java

Todos los issues de aws/aws-durable-execution-sdk-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.