[Bug Bash] ate-env delete fails and leaves the env stuck in ACTOR_STATE_DELETING

Aperta
  • Qualcosa non va?
#115 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, google-cloud
Ambito
backend, cloud

Direzione di ricerca

Start with the delete flow in Substrate's cmd/ateapi/internal/controlapi/workflow_delete.go, especially ensureExternalSnapshotsReleased, and compare its bucket access with bucketGrants in tools/setup-gcp/cmd/bucket.go. The report identifies a permissions cause and separate user-facing failures in ate-env delete and reads of deleting environments, but leaves the implementation scope across ate-env and Substrate open. Done means failed deletes and reads surface promptly, or bucket permissions are checked before a delete can fail.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug substrate

ReleaseExternalSnapshots gets a GCS 403 (storage.objects.list). Suspend wrote to the same bucket fine. Not yet confirmed: this could be IAM on my older bucket rather than a Substrate bug.

Seen in step 9 of the bug bash flow (env 40a6c63, Substrate ddb46651): ate-env delete alice failed, the env stayed in ACTOR_STATE_DELETING, and ate-env alice read /data/note.txt then waited 2 minutes and failed with Unavailable.

The bucket IAM is the cause. gs://snapshot-substrate-test-shrutinair-gke-dev was created on 2026-04-13. Its only Substrate grants are to the old atelet Workload Identity principal (roles/storage.objectAdmin, roles/storage.bucketViewer). Current tools/setup-gcp (bucketGrants in tools/setup-gcp/cmd/bucket.go) grants roles/storage.objectAdmin and roles/storage.bucketViewer on the snapshot bucket to the ate-api-server Workload Identity principal and to the snapshot transfer SA, and nothing to atelet. On this bucket, ate-api-server has no list permission. ReleaseExternalSnapshots (ensureExternalSnapshotsReleased in Substrate's cmd/ateapi/internal/controlapi/workflow_delete.go) runs as ate-api-server and gets the 403. The suspend that wrote fine went through a different identity.

The same bucket on Substrate main (ffed8551) shows the same failures: an actor stuck in ACTOR_STATE_DELETING for over 20 minutes, delete atespace failing with "not empty", and create tag failing with object-store plugin upload ... failed (Unavailable).

Workaround: re-run go run ./tools/setup-gcp create bucket (or bootstrap) from a current Substrate checkout so the bucket gets the current grants. Then remove the stale atelet grants, which give every node write access to every snapshot.

Remaining issues, in ate-env or Substrate:

  • A delete that can never succeed leaves the env in DELETING with no error shown to the user. ate-env delete should report the failure, and read on a deleting env should fail fast instead of waiting for a wake.
  • Substrate could check bucket permissions at install time (ate-setup or ate-api-server startup) instead of failing at the first delete.

Found during the Substrate GKE bug bash, with help from an AI agent.

Lingua principale
Go
Stelle
70
Fork
17
Merge medio
21h 36m
PR unite (30g)
19

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di agent-substrate/env

Tutte le issue di agent-substrate/env

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.