Stale worker claim blocks actor resume after increasing template resources

Open
  • Is something wrong?
#3,311 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
47/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
backend, databases

Research direction

Trace actor resume, stale-claim recovery, worker binding, and assignment reconciliation to find where capacity failures leave claims behind. Read the linked atomic-assignment issue #1639 for context, then follow the relevant tests for resume retries and worker capacity. Done when a failed re-bind releases the stale claim and a later resume can schedule onto a worker with enough capacity.

Written by the indexing model from the issue text.

Description

area/api-machinery area/scheduling bug-bash kind/bug

During actor resume, claiming a worker slot and updating the actor's state to RESUMING happen in two separate database transactions. If the resume fails or is interrupted between them, the actor remains SUSPENDED with a leftover worker claim.

If the actor is then updated to a template with larger CPU or memory limits than that worker can fit, every subsequent resume attempt fails:

  1. Stale-claim recovery checks whether the worker matches the template's selector and runtime constraints, without checking resource capacity, so it keeps returning the undersized worker.
  2. Binding the actor to that worker fails the capacity check and rolls back without releasing the stale claim.
  3. Resume retries against the same undersized worker until backoff is exhausted and fails with Aborted.
  4. Background assignment reconciliation ignores claims that were never recorded on the actor status, so the stale claim is never cleaned up. The actor cannot be resumed and the worker slot remains leaked.

When re-binding a recovered stale claim fails due to insufficient capacity, the stale claim should be released so the next attempt schedules the actor onto a worker with enough room. Alternatively, making worker and actor assignment atomic in a single transaction (#1639) eliminates unrecorded stale claims altogether.

Dominant language
Go
Stars
6.1k
Forks
676
Avg merge
2d 2h
Merged PRs (30d)
528

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.