Record on the actor when a resume moved it to another node, and prefer a node with the image when detaching

Aperta
  • Qualcosa non va?
#3,482 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
gcp, go, kubernetes
Ambito
backend, cloud, distributed-systems

Direzione di ricerca

The issue does not name files or tests; start by locating the resume/detach scheduling path and the actor status and atelet restore-record definitions. Trace how a resumed actor is assigned to a worker and how its source node is available. Done means detaches prefer a node that already has the image when it has room, and the actor status and restore record capture the move and source node.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area/api area/node area/scheduling kind/feature
What happens today

A suspended actor is pinned to its node. Since the detach change, if that node is full at wake time the resume moves the actor to any worker with room. That is the right behavior. Two things around it could be better.

Today Suggestion
Visibility the actor's status keeps the old node; the only sign of a move is a snapshot download in atelet's restore record record the move and the source node on the actor status
Destination chosen without looking at images, so the new node usually unpacks the image (~28 s here) prefer a node that already holds the image, the same preference #276 proposes for placement

In a density run, 36,631 of 105,400 wakes moved, and I could only count them by joining logs afterwards.

Proposal

Detach prefers a worker on a node that holds the image when one has room (see the placement proposal). The actor status records that the resume moved and from which node. atelet's restore record carries the node name.

How to see it
  1. A fleet of suspended actors that fits the slots, plus a second workload that keeps creating fresh actors so the slots on the fleet's nodes are taken between its steps.
  2. Resume the fleet. In atelet's restore records, the resumes with download above zero and seconds of oci_unpack are the ones that moved; nothing on the actor says so.
Setup

etup: two GKE worker pools. Pool A: 40 x n2-standard-8 (8 vCPU, 32 GiB, Intel Cascade Lake), 192 GB pd-balanced boot disk, 320 worker pods. Pool B: 783 x e2-standard-8 (8 vCPU, 32 GiB, AMD Rome), 500 GB pd-balanced boot disk, 2,000 worker pods. ateapi at 2 to 12 replicas; Postgres on one n2-standard-16 with pd-ssd. Images: SWE-bench Verified (445, about 1.1 GB compressed median, 13 layers), R2E-Gym-V1 (8,101, 0.57 GB compressed median, 1.9 GB max, 14 layers, almost no layer sharing), a 36,871-image SWE-RL set (0.5 to 1.4 GB compressed median per subset, up to 13 GB squashed single-layer images); unpacked size 2 to 2.5 times compressed; plain containerd pulls from a same-region Artifact Registry. gVisor (runsc), GKE 1.36.4-gke.2046002, Container-Optimized OS, Linux x86_64. Observed on Substrate 0fdb67b5; the code cited is unchanged on main at a508f8fe.

Lingua principale
Go
Stelle
5.9k
Fork
637
Merge medio
2g 1h
PR unite (30g)
523

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di agent-substrate/substrate

Tutte le issue di agent-substrate/substrate

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.