[Bug Bash] ate-env delete fails and leaves the env stuck in ACTOR_STATE_DELETING

Open
  • Is something wrong?
#115 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, google-cloud
Domain
backend, cloud

Research direction

Start with the delete flow in Substrate's cmd/ateapi/internal/controlapi/workflow_delete.go, especially ensureExternalSnapshotsReleased, and compare its bucket access with bucketGrants in tools/setup-gcp/cmd/bucket.go. The report identifies a permissions cause and separate user-facing failures in ate-env delete and reads of deleting environments, but leaves the implementation scope across ate-env and Substrate open. Done means failed deletes and reads surface promptly, or bucket permissions are checked before a delete can fail.

Written by the indexing model from the issue text.

Description

bug substrate

ReleaseExternalSnapshots gets a GCS 403 (storage.objects.list). Suspend wrote to the same bucket fine. Not yet confirmed: this could be IAM on my older bucket rather than a Substrate bug.

Seen in step 9 of the bug bash flow (env 40a6c63, Substrate ddb46651): ate-env delete alice failed, the env stayed in ACTOR_STATE_DELETING, and ate-env alice read /data/note.txt then waited 2 minutes and failed with Unavailable.

The bucket IAM is the cause. gs://snapshot-substrate-test-shrutinair-gke-dev was created on 2026-04-13. Its only Substrate grants are to the old atelet Workload Identity principal (roles/storage.objectAdmin, roles/storage.bucketViewer). Current tools/setup-gcp (bucketGrants in tools/setup-gcp/cmd/bucket.go) grants roles/storage.objectAdmin and roles/storage.bucketViewer on the snapshot bucket to the ate-api-server Workload Identity principal and to the snapshot transfer SA, and nothing to atelet. On this bucket, ate-api-server has no list permission. ReleaseExternalSnapshots (ensureExternalSnapshotsReleased in Substrate's cmd/ateapi/internal/controlapi/workflow_delete.go) runs as ate-api-server and gets the 403. The suspend that wrote fine went through a different identity.

The same bucket on Substrate main (ffed8551) shows the same failures: an actor stuck in ACTOR_STATE_DELETING for over 20 minutes, delete atespace failing with "not empty", and create tag failing with object-store plugin upload ... failed (Unavailable).

Workaround: re-run go run ./tools/setup-gcp create bucket (or bootstrap) from a current Substrate checkout so the bucket gets the current grants. Then remove the stale atelet grants, which give every node write access to every snapshot.

Remaining issues, in ate-env or Substrate:

  • A delete that can never succeed leaves the env in DELETING with no error shown to the user. ate-env delete should report the failure, and read on a deleting env should fail fast instead of waiting for a wake.
  • Substrate could check bucket permissions at install time (ate-setup or ate-api-server startup) instead of failing at the first delete.

Found during the Substrate GKE bug bash, with help from an AI agent.

Dominant language
Go
Stars
70
Forks
17
Avg merge
21h 36m
Merged PRs (30d)
19

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/env

All issues in agent-substrate/env

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.