ate-setup delete ate-system + deploy ate-system regenerates the pod-identity CA and breaks mTLS for existing client pods until their leaf certs rotate (~23h), and leaks cluster-scoped resources on --kind / --setup-csi

Open
  • Is something wrong?
#3,273 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, infrastructure, security

Research direction

Read cmd/ate-setup/internal/steps/delete.go alongside prereqs.go, deploy.go, and csi.go to trace what deploy installs and delete removes; compare the --kind kustomization with sandboxconfig-validation.yaml. Start by reproducing the CA replacement and checking which resources survive deletion. Done means existing clients retain valid mTLS during redeploy and delete removes the cluster scoped admission and CSI resources it installs.

Written by the indexing model from the issue text.

Description

area/dev-infra area/security kind/bug
Summary & Root Cause

mTLS break. DeleteAteSystem deletes and waits on the podcertificate-controller-system namespace, which destroys Secret/pod-identity-ca-pool. On the next deploy ate-system, EnsurePodCertificateCAs finds the Secret missing and generates a new CA, and the controller publishes a new ClusterTrustBundle. Kubelet refreshes projected clusterTrustBundle volumes promptly, so a pre-existing client pod trusts the new router cert. It only reissues a projected podCertificate leaf near its refreshBefore time, though (about 23h into the 24h lifetime). Until then, the client presents a leaf signed by the deleted CA, and atenet-router rejects it with unknown ca.

Kind / CSI leaks (static). On --kind, DeleteAteSystem deletes only the rendered kind kustomization plus pod-certificate-controller.yaml. DeployAteSystem also applies sandboxconfig-validation.yaml through DeploySandboxConfig, and that file is not in the kind kustomization. So the cluster-scoped ValidatingAdmissionPolicy/sandboxconfig-assets and ValidatingAdmissionPolicyBinding/sandboxconfig-assets are never deleted. Nothing in DeleteAteSystem removes what SetupCSI installs (--setup-csi=hostpath,nfs).

Affected Code Locations
  • cmd/ate-setup/internal/steps/delete.go:44-94 (DeleteAteSystem)
  • cmd/ate-setup/internal/steps/delete.go:90 (waits for namespace podcertificate-controller-system to be deleted, which removes the CA Secrets in it)
  • cmd/ate-setup/internal/steps/delete.go:51-63 (--kind branch: deletes only the kind kustomization and pod-certificate-controller.yaml)
  • cmd/ate-setup/internal/steps/delete.go:64 (non-kind branch: deletes the whole manifest directory)
  • cmd/ate-setup/internal/steps/prereqs.go:90-101 (EnsurePodCertificateCAs: creates new CA pools when pod-identity-ca-pool is missing)
  • cmd/ate-setup/internal/steps/create.go:40 (SecretPodIdentityCA = "pod-identity-ca-pool"), cmd/ate-setup/internal/steps/env.go:63 (NamespacePodCert = "podcertificate-controller-system")
  • cmd/ate-setup/internal/steps/deploy.go:119, 593-609 (DeploySandboxConfig applies sandboxconfig-validation.yaml)
  • manifests/ate-install/sandboxconfig-validation.yaml:21-23, 53-57 (ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding sandboxconfig-assets)
  • manifests/ate-install/kind/kustomization.yaml:29-39 (does not list sandboxconfig-validation.yaml)
  • cmd/ate-setup/internal/steps/csi.go:116 (SetupCSI; delete.go has no CSI cleanup)
Detailed Reproduction Guide

  1. Have a client pod (test-client) with a projected podCertificate (signerName: podidentity.podcert.ate.dev/identity) and a clusterTrustBundle volume, and confirm it reaches an actor through atenet-router.

  2. Record the CA and the client's leaf issuer:

    kubectl get secret -n podcertificate-controller-system pod-identity-ca-pool -o jsonpath='{.metadata.uid} {.metadata.creationTimestamp}'
kubectl exec -n test-client test-client -- sh -c 'openssl x509 -noout -issuer -enddate < /run/podidentity/credential-bundle.pem'

  3. Reinstall and recreate an actor:

    go run ./cmd/ate-setup delete ate-system
go run ./cmd/ate-setup deploy ate-system
go run ./cmd/ate-setup deploy demo counter
kubectl ate create actor s6-b -a ate-demo-counter --template counter

  4. Curl from the old pod and from a new pod with an identical spec:

    kubectl exec -n test-client test-client -- curl -sS -w "\nHTTP_STATUS=%{http_code}\n" \
  --cacert /run/servicedns-ca/trust-bundle.pem \
  --cert /run/podidentity/credential-bundle.pem \
  --key /run/podidentity/credential-bundle.pem \
  -H "ate-target-actor: ate-demo-counter/s6-b" \
  https://atenet-router.ate-system.svc/

  5. Kind only (static, not run on GKE):

    go run ./cmd/ate-setup deploy ate-system --kind
go run ./cmd/ate-setup delete ate-system --kind
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding | grep sandboxconfig-assets
Expected vs. Actual Behavior
  • Expected: Redeploying ate-system does not silently break mTLS for existing clients. Either the pod-identity CA is preserved across delete/deploy, the new trust bundle carries both CAs during a rollover, or the operator is told to recreate client pods. delete ate-system removes every cluster-scoped object that deploy ate-system created.
  • Actual: The old client pod fails with curl: (56) ... tlsv1 alert unknown ca, while a fresh pod with the same spec succeeds. The outage lasts until the old leaf is refreshed (inferred ~23h) or the pod is recreated. On --kind, the two sandboxconfig-assets admission objects are left behind (static).
Observed Output & Logs
  # pre-existing test-client (uid 0d2155a7):
  curl: (56) OpenSSL SSL_read: ... tlsv1 alert unknown ca
  # new test-client2 with identical spec:
  HTTP_STATUS=200
Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.