scheduler doesn't count suspended actors that are still bound to a node
Maintainers usually reply within 1 day
@BenTheElder is already working on this.
Since Oct 31, 2026.
Assessment
This issue has not been assessed yet.
Description
A suspended actor stays bound to the node that holds its local snapshot, and its next resume is required to land there.
But when placing a new actor, the scheduler only compares what workers have allocated right now, which is just the actors currently assigned ...
So for workloads that spend most of their time suspended, e.g. agents waiting on an LLM between steps, every worker looks empty, placement is effectively random, and then it sticks for the life of the actor.
We hit this in the burn-in for #3485: agentsession at 6 users on 4 workers, 6 actors living the whole 45m run. One worker got no actors in 4 of 7 runs, and another got 3. Placing 6 actors on 4 workers at random leaves one empty ~62% of the time, so this is basically random. The node with 3 actors is the one that fell behind on disk.
With the actors kept running instead, 8 actors resumed 1s apart spread 3/2/2/1, so placement works when the actors are running.
We should count actors bound to a node while suspended when picking the less loaded worker, without counting them against capacity, so they don't block placement. Letting a resume move off a busy node once the snapshot is durable is another option, but it gives up the local resume.
I searched for related issues but none are quite this: #589 (the same pin, from the capacity side), #3482 (a resume only moves when the pinned node is full, which a worker without limits never is, see #3479)
- Dominant language
- Go
- Stars
- 5.9k
- Forks
- 637
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 523
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from agent-substrate/substrate
-
Clean up the ko retry testsPossibly taken A pull request linked to this issue is open or already merged. Openarea/dev-infra area/reliability area/tests kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
agent-substrate/substrate#3461 ·
Maintainers usually reply within 1 day
-
egress: drop the empty-provider check in providerCredentialPossibly taken @cloudonly claimed this 3 days ago. Openarea/network kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
agent-substrate/substrate#3428 ·
Maintainers usually reply within 1 day
-
kubectl-ate create actor-template -f <file> fails with useless errorPossibly taken @cloudonly claimed this 4 days ago. Openarea/cli kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
agent-substrate/substrate#3406 ·
Maintainers usually reply within 1 day
-
k8s-credential-provider manifest is missing ate-otel-config envFrom, failing OTLP metric exports to [::1]:4317 / 127.0.0.1:4317 every 60sPossibly taken @ericcurtin claimed this 5 days ago. Openarea/dev-infra area/observability area/security kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
agent-substrate/substrate#3282 ·
Maintainers usually reply within 1 day
-
filecachegc.runPass suppresses eviction logs when FreedBytes < target and consecutiveShortfalls > 3Possibly taken @ericcurtin claimed this 4 days ago. Openarea/node area/observability area/storage kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
agent-substrate/substrate#3281 ·
Maintainers usually reply within 1 day
All issues in agent-substrate/substrate
Similar issues
-
fix(scan): validate and normalize output format in validate-contractPossibly taken @WKr1shna claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
bug good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Won't Fix
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
microsoft/TypeScript#65107 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
argoproj/argo-workflows#17397 ·
Maintainers usually reply within 1 day