Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Feat]: Cluster mode: detect and recover tasks abandoned by a crashed replica (heartbeat/lease)

オープン
  • 何か問題がありますか？
#1,324 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

@rohityan がすでに取り組んでいます。

2026年10月7日 から。

評価

この issue はまだ評価されていません。

説明

Is your feature request related to a problem? Please describe.

Cluster mode (#1281) handles replicas that are alive and disagree: version checks on saves, cancel from any replica, resubscribe from any replica. It does nothing for a replica that dies while running execute() (OOM kill, node loss, a rolling deploy without enough grace time).

When that happens, the task stays non-terminal forever:

  • GetTask returns TASK_STATE_WORKING indefinitely.
  • Every SubscribeToTask on another replica keeps polling task_events and never ends, because no terminal event will ever be written.
  • Nothing ever re-runs or fails the task. The only way out is a client-issued CancelTask.
  • A push-only client never hears anything again.

The client can't tell a dead replica from a slow agent. status.timestamp only moves when the agent emits an event, so a long LLM or tool call looks exactly like a crash.

Detecting abandoned tasks was a stated goal of the Go SDK's distributed mode (a2a-go discussion #139: "Supporting abandoned task detection (e.g. on process reboot)"), and a2a-go ships it (a2a-go #115, a2asrv/workqueue). The Python design discussion (#1224) doesn't cover it.

Describe the solution you'd like

An optional extension point for cluster mode, similar to a2a-go's workqueue.Heartbeater / LeaseManager, along these lines:

  1. Lease while executing. While execute() runs, the replica renews a lease (task_id, owner, expires_at, plus the task's owner scope) on an interval.
  2. Release when the turn ends. Release the lease when the task moves into a terminal or interrupted state, so a task waiting on INPUT_REQUIRED / AUTH_REQUIRED is never treated as abandoned.
  3. Sweep. On every replica, a sweeper atomically claims leases that have expired (for example FOR UPDATE SKIP LOCKED, so no leader is needed) and applies a policy:
    • fail (default): write TASK_STATE_FAILED;
    • re-execute: run the turn again, with an attempt counter.
  4. Write the failure through VersionedTaskStore.save(..., event=...). This matters in cluster mode:
    • the event row ends every remote SubscribeToTask stream with FAILED;
    • the version bump fences a replica that wasn't really dead (a long GC pause or a network partition). Its next save gets ConcurrentTaskModificationError, it re-reads, sees FAILED, and cancels its producer, the same way a remote cancel works today;
    • push senders can then notify FAILED.

Pitfalls we hit running a heartbeat plus sweeper outside the SDK. They're probably worth designing in from the start:

  • Release after the durable save, not when execute() returns. The terminal or interrupted status event may still be on the event queue when execute() returns. A crash in that gap leaves a non-terminal task with no lease, which no sweep can ever find.
  • Release on a state change, not on whatever state is being saved. On a resumed turn, the first save is the held user-message flush, and it re-saves the stale INPUT_REQUIRED state (update_with_message doesn't change status.state). Releasing on "saved an interrupted state" drops the lease at the start of the resumed turn.
  • A renewal must not recreate a released lease. If renewal is an upsert, a heartbeat that lands just after the release brings the lease back. The sweeper later claims it and fails a task that was correctly paused.
  • The sweeper needs the owner scope. DatabaseTaskStore / VersionedDatabaseTaskStore filter reads by owner_resolver(context). A sweeper calling get() with an empty ServerCallContext gets None for authenticated tasks and silently does nothing. Store the owner scope with the lease and rebuild the context from it.
Describe alternatives you've considered
  • Handle it in the application (what we do today): heartbeat rows, an orphan sweeper, and a TaskStore wrapper that releases the lease after the durable save. It works, but every deployment has to rebuild it. In cluster mode it also needs the versioned save path and the event table to end remote streams correctly, and those are SDK internals an application shouldn't have to coordinate with.
  • Leave it out of scope, as a2a-rs did (a2a-rs#149). That's reasonable for single-process servers. But cluster mode is aimed specifically at multi-replica deployments, and replicas dying is the normal failure in that setting.
  • A client-side overall timeout is a useful backstop, but the client can't tell which tasks are actually dead.
  • The spec-level "Expectations" proposal (A2A#2209) is about business deadlines, not process liveness.
Additional context
  • Defaults that worked for us: heartbeat every 30s, a task counts as abandoned after 120s without one, and the sweep runs every 30s. That's roughly 2–2.5 minutes to detect a dead replica.
  • Happy to share more detail or help with an implementation if the maintainers think this belongs in the SDK.
Code of Conduct
  • I agree to follow this project's Code of Conduct
主要言語
Python
スター
2.2k
フォーク
509
平均マージ
3日 18時間
マージ済み PR（30日）
44

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

a2aproject/a2a-python のほかの issue

a2aproject/a2a-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。