Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Feat]: Cluster mode: detect and recover tasks abandoned by a crashed replica (heartbeat/lease)

Abierto
  • ¿Algo no está bien?
#1,324 4 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 2 días

@ishymko ya está trabajando en esto.

Desde el 9/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

component: server status: needs review
Is your feature request related to a problem? Please describe.

Cluster mode (#1281) handles replicas that are alive and disagree: version checks on saves, cancel from any replica, resubscribe from any replica. It does nothing for a replica that dies while running execute() (OOM kill, node loss, a rolling deploy without enough grace time).

When that happens, the task stays non-terminal forever:

  • GetTask returns TASK_STATE_WORKING indefinitely.
  • Every SubscribeToTask on another replica keeps polling task_events and never ends, because no terminal event will ever be written.
  • Nothing ever re-runs or fails the task. The only way out is a client-issued CancelTask.
  • A push-only client never hears anything again.

The client can't tell a dead replica from a slow agent. status.timestamp only moves when the agent emits an event, so a long LLM or tool call looks exactly like a crash.

Detecting abandoned tasks was a stated goal of the Go SDK's distributed mode (a2a-go discussion #139: "Supporting abandoned task detection (e.g. on process reboot)"), and a2a-go ships it (a2a-go #115, a2asrv/workqueue). The Python design discussion (#1224) doesn't cover it.

Describe the solution you'd like

An optional extension point for cluster mode, similar to a2a-go's workqueue.Heartbeater / LeaseManager, along these lines:

  1. Lease while executing. While execute() runs, the replica renews a lease (task_id, owner, expires_at, plus the task's owner scope) on an interval.
  2. Release when the turn ends. Release the lease when the task moves into a terminal or interrupted state, so a task waiting on INPUT_REQUIRED / AUTH_REQUIRED is never treated as abandoned.
  3. Sweep. On every replica, a sweeper atomically claims leases that have expired (for example FOR UPDATE SKIP LOCKED, so no leader is needed) and applies a policy:
    • fail (default): write TASK_STATE_FAILED;
    • re-execute: run the turn again, with an attempt counter.
  4. Write the failure through VersionedTaskStore.save(..., event=...). This matters in cluster mode:
    • the event row ends every remote SubscribeToTask stream with FAILED;
    • the version bump fences a replica that wasn't really dead (a long GC pause or a network partition). Its next save gets ConcurrentTaskModificationError, it re-reads, sees FAILED, and cancels its producer, the same way a remote cancel works today;
    • push senders can then notify FAILED.

Pitfalls we hit running a heartbeat plus sweeper outside the SDK. They're probably worth designing in from the start:

  • Release after the durable save, not when execute() returns. The terminal or interrupted status event may still be on the event queue when execute() returns. A crash in that gap leaves a non-terminal task with no lease, which no sweep can ever find.
  • Release on a state change, not on whatever state is being saved. On a resumed turn, the first save is the held user-message flush, and it re-saves the stale INPUT_REQUIRED state (update_with_message doesn't change status.state). Releasing on "saved an interrupted state" drops the lease at the start of the resumed turn.
  • A renewal must not recreate a released lease. If renewal is an upsert, a heartbeat that lands just after the release brings the lease back. The sweeper later claims it and fails a task that was correctly paused.
  • The sweeper needs the owner scope. DatabaseTaskStore / VersionedDatabaseTaskStore filter reads by owner_resolver(context). A sweeper calling get() with an empty ServerCallContext gets None for authenticated tasks and silently does nothing. Store the owner scope with the lease and rebuild the context from it.
Describe alternatives you've considered
  • Handle it in the application (what we do today): heartbeat rows, an orphan sweeper, and a TaskStore wrapper that releases the lease after the durable save. It works, but every deployment has to rebuild it. In cluster mode it also needs the versioned save path and the event table to end remote streams correctly, and those are SDK internals an application shouldn't have to coordinate with.
  • Leave it out of scope, as a2a-rs did (a2a-rs#149). That's reasonable for single-process servers. But cluster mode is aimed specifically at multi-replica deployments, and replicas dying is the normal failure in that setting.
  • A client-side overall timeout is a useful backstop, but the client can't tell which tasks are actually dead.
  • The spec-level "Expectations" proposal (A2A#2209) is about business deadlines, not process liveness.
Additional context
  • Defaults that worked for us: heartbeat every 30s, a task counts as abandoned after 120s without one, and the sweep runs every 30s. That's roughly 2–2.5 minutes to detect a dead replica.
  • Happy to share more detail or help with an implementation if the maintainers think this belongs in the SDK.
Code of Conduct
  • I agree to follow this project's Code of Conduct
Lenguaje dominante
Python
Estrellas
2.2k
Forks
509
Merge medio
3 d 11 h
PR fusionados (30 d)
43

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de a2aproject/a2a-python

Todos los issues de a2aproject/a2a-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.