Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

[Feat]: Cluster mode: detect and recover tasks abandoned by a crashed replica (heartbeat/lease)

Ouverte
  • Quelque chose ne va pas ?
#1,324 5 commentaires 0 réactions 1 personne assignée Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

@mykytanetipa y travaille déjà.

Depuis le 18/10/2026.

Évaluation

Cette issue n'a pas encore été évaluée.

Description

component: server maintainers-only
Is your feature request related to a problem? Please describe.

Cluster mode (#1281) handles replicas that are alive and disagree: version checks on saves, cancel from any replica, resubscribe from any replica. It does nothing for a replica that dies while running execute() (OOM kill, node loss, a rolling deploy without enough grace time).

When that happens, the task stays non-terminal forever:

  • GetTask returns TASK_STATE_WORKING indefinitely.
  • Every SubscribeToTask on another replica keeps polling task_events and never ends, because no terminal event will ever be written.
  • Nothing ever re-runs or fails the task. The only way out is a client-issued CancelTask.
  • A push-only client never hears anything again.

The client can't tell a dead replica from a slow agent. status.timestamp only moves when the agent emits an event, so a long LLM or tool call looks exactly like a crash.

Detecting abandoned tasks was a stated goal of the Go SDK's distributed mode (a2a-go discussion #139: "Supporting abandoned task detection (e.g. on process reboot)"), and a2a-go ships it (a2a-go #115, a2asrv/workqueue). The Python design discussion (#1224) doesn't cover it.

Describe the solution you'd like

An optional extension point for cluster mode, similar to a2a-go's workqueue.Heartbeater / LeaseManager, along these lines:

  1. Lease while executing. While execute() runs, the replica renews a lease (task_id, owner, expires_at, plus the task's owner scope) on an interval.
  2. Release when the turn ends. Release the lease when the task moves into a terminal or interrupted state, so a task waiting on INPUT_REQUIRED / AUTH_REQUIRED is never treated as abandoned.
  3. Sweep. On every replica, a sweeper atomically claims leases that have expired (for example FOR UPDATE SKIP LOCKED, so no leader is needed) and applies a policy:
    • fail (default): write TASK_STATE_FAILED;
    • re-execute: run the turn again, with an attempt counter.
  4. Write the failure through VersionedTaskStore.save(..., event=...). This matters in cluster mode:
    • the event row ends every remote SubscribeToTask stream with FAILED;
    • the version bump fences a replica that wasn't really dead (a long GC pause or a network partition). Its next save gets ConcurrentTaskModificationError, it re-reads, sees FAILED, and cancels its producer, the same way a remote cancel works today;
    • push senders can then notify FAILED.

Pitfalls we hit running a heartbeat plus sweeper outside the SDK. They're probably worth designing in from the start:

  • Release after the durable save, not when execute() returns. The terminal or interrupted status event may still be on the event queue when execute() returns. A crash in that gap leaves a non-terminal task with no lease, which no sweep can ever find.
  • Release on a state change, not on whatever state is being saved. On a resumed turn, the first save is the held user-message flush, and it re-saves the stale INPUT_REQUIRED state (update_with_message doesn't change status.state). Releasing on "saved an interrupted state" drops the lease at the start of the resumed turn.
  • A renewal must not recreate a released lease. If renewal is an upsert, a heartbeat that lands just after the release brings the lease back. The sweeper later claims it and fails a task that was correctly paused.
  • The sweeper needs the owner scope. DatabaseTaskStore / VersionedDatabaseTaskStore filter reads by owner_resolver(context). A sweeper calling get() with an empty ServerCallContext gets None for authenticated tasks and silently does nothing. Store the owner scope with the lease and rebuild the context from it.
Describe alternatives you've considered
  • Handle it in the application (what we do today): heartbeat rows, an orphan sweeper, and a TaskStore wrapper that releases the lease after the durable save. It works, but every deployment has to rebuild it. In cluster mode it also needs the versioned save path and the event table to end remote streams correctly, and those are SDK internals an application shouldn't have to coordinate with.
  • Leave it out of scope, as a2a-rs did (a2a-rs#149). That's reasonable for single-process servers. But cluster mode is aimed specifically at multi-replica deployments, and replicas dying is the normal failure in that setting.
  • A client-side overall timeout is a useful backstop, but the client can't tell which tasks are actually dead.
  • The spec-level "Expectations" proposal (A2A#2209) is about business deadlines, not process liveness.
Additional context
  • Defaults that worked for us: heartbeat every 30s, a task counts as abandoned after 120s without one, and the sweep runs every 30s. That's roughly 2–2.5 minutes to detect a dead replica.
  • Happy to share more detail or help with an implementation if the maintainers think this belongs in the SDK.
Code of Conduct
  • I agree to follow this project's Code of Conduct
Langage dominant
Python
Étoiles
2.2k
Forks
518
Merge moyen
3 j 20 h
PR mergées (30 j)
42

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de a2aproject/a2a-python

Toutes les issues de a2aproject/a2a-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.