[bug-bash] After a servicedns CA rotation, actor suspends crash on nodes whose atelet predates the new CA (objectstore-plugin loads exchange CA bundle once)

Read internal/objectstoreplugin/exchangeclient/exchangeclient.go:89-92 and compare its CA loading with the credbundle.PoolLoader use in the sites fixed for #1931. Check whether the exchange client picks up trust bundle changes during later TLS handshakes, and inspect relevant tests or add a focused regression test. Done means suspending an actor succeeds after CA rotation without restarting atelet, with the failure explained by useful logs.

What happened?

When rotating podcertificate-controller-system/service-dns-ca-pool using the 3-step zero-downtime procedure documented in internal/localca/localca.go:50-59 (add new inactive CA -> wait for ClusterTrustBundle and projected pod volume propagation -> activate new CA -> ate-api-server receives a leaf signed by the new CA), suspending any actor that requires a fresh token exchange on a node whose atelet pod started before the new CA was added fails with Unavailable during external snapshot upload and crashes the actor (ACTOR_STATE_CRASHED), losing its unsnapshotted state.

• Restarting atelet on one node (node-sznr) immediately restores suspend for actors placed on that node (config-ca4, config-ca5), while actors placed on the un-restarted node (node-02fq: config-ca2, config-ca3) continue to crash until that node's atelet is also restarted.
• Zero log lines are emitted by objectstore-plugin on the node or by the :8445 exchange on ate-api-server explaining the TLS verification failure.

Expected Behavior

Per internal/localca/localca.go:15-30, 50-59 and internal/credbundle/credbundle.go:56-65 ("a pool built at startup never sees a CA rotation"):

• Once the projected /run/servicedns.podcert.ate.dev/trust-bundle.pem on each node contains both the old and new CA roots, activating the new CA for signing requires no atelet restarts.
• kubectl ate suspend actor <name> -a <atespace> --mode=sync succeeds and transitions the actor to ACTOR_STATE_SUSPENDED.

Steps to Reproduce

1. Follow the 3-step rotation procedure in internal/localca/localca.go:50-59 on podcertificate-controller-system/service-dns-ca-pool:
   • Step 1 (Add inactive CA 2): Add a second CA entry (id: "2") to pool in Secret podcertificate-controller-system/service-dns-ca-pool while keeping CA 1 as activeForSigning.
   • Step 2 (Wait for trust propagation): Wait ~2 minutes until ClusterTrustBundle servicedns.podcert.ate.dev:identity:primary-bundle (~64s) and projected /run/servicedns.podcert.ate.dev/trust-bundle.pem files in running atelet pods (~128s) contain both CA 1 and CA 2 roots.
   • Step 3 (Activate CA 2 & issue new serving leaf): Set activeForSigning to "2" in service-dns-ca-pool, and either wait for ate-api-server's servicedns cert to renew or run:

     kubectl -n ate-system rollout restart deploy/ate-api-server
     kubectl -n ate-system rollout status deploy/ate-api-server
     

2. Create a fresh actor, wake it via the router, and run a synchronous suspend:

   kubectl ate create actor config-ca1 -a ate-demo-counter --template counter
   ate_curl config-ca1
   kubectl ate suspend actor config-ca1 -a ate-demo-counter --mode=sync
   kubectl ate get actor config-ca1 -a ate-demo-counter
   

3. A/B node control: Restart the atelet pod on one of the two nodes (node-sznr) while leaving atelet untouched on the other node (node-02fq), then create, wake, and suspend fresh actors on each node (config-ca2/config-ca3 on node-02fq, config-ca4/config-ca5 on node-sznr).

Sandbox Runtime

gVisor (runsc)

Agent Substrate Version / Commit SHA

ffed8551

Kubernetes Version & Environment

1.37.1-gke.2097000

Host OS & Architecture

Linux x86_64 (COS, running on GKE)

Relevant Logs and Diagnostic Output

$ kubectl ate suspend actor config-ca1 -a ate-demo-counter --mode=sync
Error: failed to suspend actor: rpc error: code = Internal desc = internal server error: workflow failed at step CallAteletSuspend:
actor ate-demo-counter/config-ca1 crashed: rpc error: code = Internal desc = while uploading external snapshot: while uploading
snapshot files: rpc error: code = Unavailable desc = object-store plugin upload of gs://ate-snapshots-maxthomp-bugbash-cfg/atespaces/
ate-demo-counter/actors/8d0111f8-.../snapshots/939b4a6d-... failed (Unavailable)

== A/B node control: restart only atelet-fh8tj (node sznr); leave atelet-9gx67 (node 02fq) running
pod "atelet-fh8tj" deleted from ate-system namespace
NAME           READY   STATUS    RESTARTS   AGE   IP           NODE
atelet-5wn6v   2/2     Running   0          12s   10.96.1.28   gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-sznr
atelet-9gx67   2/2     Running   0          47m   10.96.0.21   gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-02fq
21:22:58 config-ca2 node=gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-02fq suspend: Error: failed to suspend actor: rpc error: code = Internal desc = internal server error: workflow failed at step CallAteletSuspend: actor ate-demo-counter/config-ca2 crashed: rpc error: code = Internal ...
21:23:00 config-ca3 node=gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-02fq suspend: Error: failed to suspend actor: rpc error: code = Internal desc = internal server error: workflow failed at step CallAteletSuspend: actor ate-demo-counter/config-ca3 crashed: rpc error: code = Internal ...
21:23:02 config-ca4 node=gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-sznr suspend: ate-demo-counter   config-ca4   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>   5   1s
21:23:04 config-ca5 node=gke-bugbash-maxthomp-substrate-node-p-2ec97ad4-sznr suspend: ate-demo-counter   config-ca5   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>   5   1s

Additional Context

Root Cause:
internal/objectstoreplugin/exchangeclient/exchangeclient.go:89-92 loads the exchange trust bundle once at process startup via credbundle.ParsePool(cfg.CAFile) and freezes that x509.CertPool into a static tls.Config on a long-lived grpc.ClientConn, instead of reloading per handshake via credbundle.PoolLoader:

roots, err := credbundle.ParsePool(cfg.CAFile)
if err != nil {
    return nil, fmt.Errorf("load exchange CA bundle: %w", err)
}

Relationship to #1931:
Open issue #1931 tracks the same bug class for cmd/atelet/main.go, internal/ateapiauth/client.go, and internal/ateletdial/ateletdial.go. At ffed8551, all 3 of those sites listed in #1931 have already been fixed to use credbundle.PoolLoader, whereas the following 3 sites in main are not listed in #1931 and still load the CA bundle once:

• internal/objectstoreplugin/exchangeclient/exchangeclient.go:89-92 (added in #2762)
• internal/atunnel/egress/client.go:115-121 (os.ReadFile(cfg.TrustBundlePath) + AppendCertsFromPEM once per NewClient on actor activation)
• cmd/ateapi/internal/controlapi/dialer.go:200-205 (x509bundle.Load once per cached atelet connection in newVerifyConnection)

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.