ActorTemplate golden snapshot (`GOLDEN_SNAPSHOT_MODE_ENABLED`) never refreshes on SandboxConfig runtime upgrade, pinning new actors to the old runtime or degrading them to ROOTFS cold boot

Open
  • Is something wrong?
#3,275 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, infrastructure

Research direction

Start in cmd/ateapi/internal/controlapi/template_reconciler.go, especially the periodic scan and goldenSnapshotDone, to trace how a terminal golden status prevents rebuilding after defaultVersion changes. Then follow how actor.go assigns the golden tag and how workflow_resume.go handles an unavailable runtime. Done means a template golden snapshot is rebuilt for the current runtime and new actors restore from it; add or update tests for both runtime-upgrade cases.

Written by the indexing model from the issue text.

Description

area/api-machinery area/scheduling area/storage kind/bug
Summary & Root Cause

The template reconciler treats "a golden tag exists" as permanently done. The golden tag records no sandbox runtime, and nothing compares the tag's runtime with the pool's current defaultVersion. A SandboxConfig upgrade therefore never re-queues the template (template_reconciler.go:133) or rebuilds the snapshot (:225). Every new actor copies the old tag at create time (actor.go:94). On first wake:

  • If the old runtime is still Enabled, it is in the worker's restorableRuntimes. The scheduler restores the actor at MEMORY on the old runtime, and every later checkpoint stays on that runtime.
  • If the old runtime is Disabled, no worker can restore it, so the resume falls back to a fresh boot without memory state (workflow_resume.go:883).
Affected Code Locations
  • cmd/ateapi/internal/controlapi/template_reconciler.go:128-138: the periodic scan skips any template whose golden status is terminal
  • cmd/ateapi/internal/controlapi/template_reconciler.go:219-228: the reconcile loop returns as soon as goldenSnapshotStatus.GetGoldenTag() != nil
  • cmd/ateapi/internal/controlapi/template_reconciler.go:418-422: goldenSnapshotDone returns GetGoldenTag() != nil || GetErrorMessage() != ""
  • cmd/ateapi/internal/controlapi/actor.go:91-94: CreateActor freezes template.GetStatus().GetGoldenSnapshotStatus().GetGoldenTag() as the new actor's source tag
  • cmd/ateapi/internal/controlapi/workflow_resume.go:879-888: logRuntimeFallback logs the fallback when the assigned worker cannot restore the snapshot's runtime
// goldenSnapshotDone reports whether the golden snapshot build reached a
// terminal state: the snapshot was recorded, or the build failed.
func goldenSnapshotDone(snapshotStatus *ateapi.GoldenSnapshotStatus) bool {
	return snapshotStatus.GetGoldenTag() != nil || snapshotStatus.GetErrorMessage() != ""
}
Detailed Reproduction Guide
  1. Deploy the counter demo (its template defaults to GOLDEN_SNAPSHOT_MODE_ENABLED) while SandboxConfig/gvisor-default has defaultVersion: v5, and wait for the golden tag:
go run ./cmd/ate-setup deploy demo counter
kubectl ate get actor-templates -a ate-demo-counter
  1. Case A: set defaultVersion: v6 and keep v5 Enabled. Wait until workers advertise default_runtime: v6 and restorable_runtimes: [v5], then create and wake a new actor:
kubectl patch sandboxconfig gvisor-default --type=merge -p '{"spec":{"defaultVersion":"v6"}}'
kubectl ate create actor s8-case-a --template counter -a ate-demo-counter
kubectl ate resume actor s8-case-a -a ate-demo-counter
  1. Case B: set v5 to state: Disabled (the JSON patch path is the index of v5 in spec.versions). Wait until workers advertise only v6, then create and wake another actor:
kubectl patch sandboxconfig gvisor-default --type=json -p '[{"op":"replace","path":"/spec/versions/0/state","value":"Disabled"}]'
kubectl ate create actor s8-v5-disabled-3 --template counter -a ate-demo-counter
kubectl ate resume actor s8-v5-disabled-3 -a ate-demo-counter
kubectl logs -n ate-system deploy/ate-api-server | grep 'cannot restore the snapshot'
Expected vs. Actual Behavior
  • Expected: After the pool's default runtime moves from v5 to v6 (or at least once v5 is Disabled), the template's golden snapshot is rebuilt on v6. New actors then run on v6 and restore from a v6 MEMORY snapshot.
  • Actual: The golden tag never changes. In Case A, every new actor restores and runs on v5, so defaultVersion: v6 has no effect on new actors of the template. In Case B, every new actor falls back to a ROOTFS cold boot on first wake. Both persist until the ActorTemplate is deleted and recreated.
Observed Output & Logs
{"time":"2026-10-27T00:57:46.261846888Z","level":"WARN","msg":"Assigned worker cannot restore the snapshot's runtime; restoring without memory state","actor":"ate-demo-counter/s8-v5-disabled-3","worker":"counter-68d64789c8-v7qwl","required_runtime":{"sandbox_class":"gvisor","version":{"schema_version":"v1","attributes":[{"key":"architecture","value":"amd64"},{"key":"cpu_features","value":"ac0004002c100800000001210000000f00000800d19f2ffb1f8bfbfffefa3223"},{"key":"sandbox_bundle_hash","value":"sha256:078445e3672a4bad894f713a43f519e50bff3b9b09a257b8b30a1ecc6ebb5b58"}]}},"assigned_runtime":{"sandbox_class":"gvisor","version":{"schema_version":"v1","attributes":[{"key":"architecture","value":"amd64"},{"key":"cpu_features","value":"ac0004002c100800000001210000000f00000800d19f2ffb1f8bfbfffefa3223"},{"key":"sandbox_bundle_hash","value":"sha256:914c887f6c3bb4b2555c3bf9060750736fb2f3c31a9d86b90a75bd86c5a9c90d"}]}},"trace_id":"b161d08155493faccc2c06e83fef3c89","span_id":"44a4ae6c7c53128e","trace_flags":"00"}

{"tag":"18c688e9-cf06-4daf-b494-47e419ccad08","bundle":"sha256:078445e3672a4bad894f713a43f519e50bff3b9b09a257b8b30a1ecc6ebb5b58","snapTime":"2026-10-27T02:20:56.767511441Z"}
{"actor":"s8a","worker":"counter-68d64789c8-zvfc4","assigned_bundle":"sha256:078445e3672a4bad894f713a43f519e50bff3b9b09a257b8b30a1ecc6ebb5b58","seeded_from":"bff50703-2a44-4b20-a6d6-016bfea2eefa"}
{"time":"2026-10-27T02:23:16.14485632Z","msg":"Assigned worker cannot restore the snapshot's runtime; restoring without memory state","actor":"ate-demo-counter/s8b","worker":"counter-68d64789c8-x77zr"}
{"goldenTag":{"atespace":"ate-demo-counter","name":"18c688e9-cf06-4daf-b494-47e419ccad08"},"snapshotTime":"2026-10-27T02:20:56.130180800Z"}
Dominant language
Go
Stars
6.1k
Forks
676
Avg merge
2d 2h
Merged PRs (30d)
538

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.