[Feat]: Cluster mode: detect and recover tasks abandoned by a crashed replica (heartbeat/lease)
Maintainers usually reply within 2 days
Assessment
This issue has not been assessed yet.
Description
Is your feature request related to a problem? Please describe.
Cluster mode (#1281) handles replicas that are alive and disagree: version checks on saves, cancel from any replica, resubscribe from any replica. It does nothing for a replica that dies while running execute() (OOM kill, node loss, a rolling deploy without enough grace time).
When that happens, the task stays non-terminal forever:
GetTaskreturnsTASK_STATE_WORKINGindefinitely.- Every
SubscribeToTaskon another replica keeps pollingtask_eventsand never ends, because no terminal event will ever be written. - Nothing ever re-runs or fails the task. The only way out is a client-issued
CancelTask. - A push-only client never hears anything again.
The client can't tell a dead replica from a slow agent. status.timestamp only moves when the agent emits an event, so a long LLM or tool call looks exactly like a crash.
Detecting abandoned tasks was a stated goal of the Go SDK's distributed mode (a2a-go discussion #139: "Supporting abandoned task detection (e.g. on process reboot)"), and a2a-go ships it (a2a-go #115, a2asrv/workqueue). The Python design discussion (#1224) doesn't cover it.
Describe the solution you'd like
An optional extension point for cluster mode, similar to a2a-go's workqueue.Heartbeater / LeaseManager, along these lines:
- Lease while executing. While
execute()runs, the replica renews a lease (task_id,owner,expires_at, plus the task's owner scope) on an interval. - Release when the turn ends. Release the lease when the task moves into a terminal or interrupted state, so a task waiting on
INPUT_REQUIRED/AUTH_REQUIREDis never treated as abandoned. - Sweep. On every replica, a sweeper atomically claims leases that have expired (for example
FOR UPDATE SKIP LOCKED, so no leader is needed) and applies a policy:fail(default): writeTASK_STATE_FAILED;re-execute: run the turn again, with an attempt counter.
- Write the failure through
VersionedTaskStore.save(..., event=...). This matters in cluster mode:- the event row ends every remote
SubscribeToTaskstream withFAILED; - the version bump fences a replica that wasn't really dead (a long GC pause or a network partition). Its next save gets
ConcurrentTaskModificationError, it re-reads, seesFAILED, and cancels its producer, the same way a remote cancel works today; - push senders can then notify
FAILED.
- the event row ends every remote
Pitfalls we hit running a heartbeat plus sweeper outside the SDK. They're probably worth designing in from the start:
- Release after the durable save, not when
execute()returns. The terminal or interrupted status event may still be on the event queue whenexecute()returns. A crash in that gap leaves a non-terminal task with no lease, which no sweep can ever find. - Release on a state change, not on whatever state is being saved. On a resumed turn, the first save is the held user-message flush, and it re-saves the stale
INPUT_REQUIREDstate (update_with_messagedoesn't changestatus.state). Releasing on "saved an interrupted state" drops the lease at the start of the resumed turn. - A renewal must not recreate a released lease. If renewal is an upsert, a heartbeat that lands just after the release brings the lease back. The sweeper later claims it and fails a task that was correctly paused.
- The sweeper needs the owner scope.
DatabaseTaskStore/VersionedDatabaseTaskStorefilter reads byowner_resolver(context). A sweeper callingget()with an emptyServerCallContextgetsNonefor authenticated tasks and silently does nothing. Store the owner scope with the lease and rebuild the context from it.
Describe alternatives you've considered
- Handle it in the application (what we do today): heartbeat rows, an orphan sweeper, and a
TaskStorewrapper that releases the lease after the durable save. It works, but every deployment has to rebuild it. In cluster mode it also needs the versioned save path and the event table to end remote streams correctly, and those are SDK internals an application shouldn't have to coordinate with. - Leave it out of scope, as a2a-rs did (a2a-rs#149). That's reasonable for single-process servers. But cluster mode is aimed specifically at multi-replica deployments, and replicas dying is the normal failure in that setting.
- A client-side overall timeout is a useful backstop, but the client can't tell which tasks are actually dead.
- The spec-level "Expectations" proposal (A2A#2209) is about business deadlines, not process liveness.
Additional context
- Defaults that worked for us: heartbeat every 30s, a task counts as abandoned after 120s without one, and the sweep runs every 30s. That's roughly 2–2.5 minutes to detect a dead replica.
- Happy to share more detail or help with an implementation if the maintainers think this belongs in the SDK.
Code of Conduct
- I agree to follow this project's Code of Conduct
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 540
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 56
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from a2aproject/a2a-python
-
[Bug]: `A2A-Version` validation ignores minor versions and the query parameterPossibly taken @rohityan claimed this 82 days ago. Opencomponent: server status:awaiting response status:stale
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
a2aproject/a2a-python#1204 · 4 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
[Bug]: REST task/request id sanitizationMay be free again A pull request for this issue was closed without being merged. Openmaintainers-only
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
a2aproject/a2a-python#805 · 1 comment ·
Maintainers usually reply within 2 days
-
[Bug]: JSON-RPC client ignores the JSON-RPC error body when the HTTP status is not 2xxPossibly taken @rohityan claimed this today. Open
a2aproject/a2a-python#1475 · 2 assignees ·
Maintainers usually reply within 2 days
-
[Feat]: validate_input_modes: match media ranges such as image/* declared in the agent cardPossibly taken @rohityan claimed this 2 days ago. Open
a2aproject/a2a-python#1468 · 2 assignees ·
Maintainers usually reply within 2 days
-
Required extensions aren't enforced, so ExtensionSupportRequiredError is never raisedPossibly taken @rohityan claimed this 2 days ago. Opencomponent: server status:awaiting response
a2aproject/a2a-python#1467 · 1 comment · 1 assignee ·
Maintainers usually reply within 2 days
All issues in a2aproject/a2a-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Install the skills as a Claude Code pluginPossibly taken @ruban-24 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 6 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationOpendata
Difficulty 2/5 1-3 hours Newbie friendliness 66/100