scheduler doesn't count suspended actors that are still bound to a node

Abierto
  • ¿Algo no está bien?
#3,528 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@BenTheElder ya está trabajando en esto.

Desde el 31/10/2026.

  • #3586 de @BenTheElder — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

area/scheduling kind/bug

A suspended actor stays bound to the node that holds its local snapshot, and its next resume is required to land there.

But when placing a new actor, the scheduler only compares what workers have allocated right now, which is just the actors currently assigned ...

So for workloads that spend most of their time suspended, e.g. agents waiting on an LLM between steps, every worker looks empty, placement is effectively random, and then it sticks for the life of the actor.

We hit this in the burn-in for #3485: agentsession at 6 users on 4 workers, 6 actors living the whole 45m run. One worker got no actors in 4 of 7 runs, and another got 3. Placing 6 actors on 4 workers at random leaves one empty ~62% of the time, so this is basically random. The node with 3 actors is the one that fell behind on disk.

With the actors kept running instead, 8 actors resumed 1s apart spread 3/2/2/1, so placement works when the actors are running.

We should count actors bound to a node while suspended when picking the less loaded worker, without counting them against capacity, so they don't block placement. Letting a resume move off a busy node once the snapshot is durable is another option, but it gives up the local resume.

I searched for related issues but none are quite this: #589 (the same pin, from the capacity side), #3482 (a resume only moves when the pinned node is full, which a worker without limits never is, see #3479)

Lenguaje dominante
Go
Estrellas
6.1k
Forks
676
Merge medio
2 d 41 min
PR fusionados (30 d)
500

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de agent-substrate/substrate

Todos los issues de agent-substrate/substrate

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.