ate-setup delete ate-system + deploy ate-system regenerates the pod-identity CA and breaks mTLS for existing client pods until their leaf certs rotate (~23h), and leaks cluster-scoped resources on --kind / --setup-csi

Aperta
  • Qualcosa non va?
#3,273 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
backend, infrastructure, security

Direzione di ricerca

Read cmd/ate-setup/internal/steps/delete.go alongside prereqs.go, deploy.go, and csi.go to trace what deploy installs and delete removes; compare the --kind kustomization with sandboxconfig-validation.yaml. Start by reproducing the CA replacement and checking which resources survive deletion. Done means existing clients retain valid mTLS during redeploy and delete removes the cluster scoped admission and CSI resources it installs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area/dev-infra area/security kind/bug
Summary & Root Cause

mTLS break. DeleteAteSystem deletes and waits on the podcertificate-controller-system namespace, which destroys Secret/pod-identity-ca-pool. On the next deploy ate-system, EnsurePodCertificateCAs finds the Secret missing and generates a new CA, and the controller publishes a new ClusterTrustBundle. Kubelet refreshes projected clusterTrustBundle volumes promptly, so a pre-existing client pod trusts the new router cert. It only reissues a projected podCertificate leaf near its refreshBefore time, though (about 23h into the 24h lifetime). Until then, the client presents a leaf signed by the deleted CA, and atenet-router rejects it with unknown ca.

Kind / CSI leaks (static). On --kind, DeleteAteSystem deletes only the rendered kind kustomization plus pod-certificate-controller.yaml. DeployAteSystem also applies sandboxconfig-validation.yaml through DeploySandboxConfig, and that file is not in the kind kustomization. So the cluster-scoped ValidatingAdmissionPolicy/sandboxconfig-assets and ValidatingAdmissionPolicyBinding/sandboxconfig-assets are never deleted. Nothing in DeleteAteSystem removes what SetupCSI installs (--setup-csi=hostpath,nfs).

Affected Code Locations
  • cmd/ate-setup/internal/steps/delete.go:44-94 (DeleteAteSystem)
  • cmd/ate-setup/internal/steps/delete.go:90 (waits for namespace podcertificate-controller-system to be deleted, which removes the CA Secrets in it)
  • cmd/ate-setup/internal/steps/delete.go:51-63 (--kind branch: deletes only the kind kustomization and pod-certificate-controller.yaml)
  • cmd/ate-setup/internal/steps/delete.go:64 (non-kind branch: deletes the whole manifest directory)
  • cmd/ate-setup/internal/steps/prereqs.go:90-101 (EnsurePodCertificateCAs: creates new CA pools when pod-identity-ca-pool is missing)
  • cmd/ate-setup/internal/steps/create.go:40 (SecretPodIdentityCA = "pod-identity-ca-pool"), cmd/ate-setup/internal/steps/env.go:63 (NamespacePodCert = "podcertificate-controller-system")
  • cmd/ate-setup/internal/steps/deploy.go:119, 593-609 (DeploySandboxConfig applies sandboxconfig-validation.yaml)
  • manifests/ate-install/sandboxconfig-validation.yaml:21-23, 53-57 (ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding sandboxconfig-assets)
  • manifests/ate-install/kind/kustomization.yaml:29-39 (does not list sandboxconfig-validation.yaml)
  • cmd/ate-setup/internal/steps/csi.go:116 (SetupCSI; delete.go has no CSI cleanup)
Detailed Reproduction Guide

  1. Have a client pod (test-client) with a projected podCertificate (signerName: podidentity.podcert.ate.dev/identity) and a clusterTrustBundle volume, and confirm it reaches an actor through atenet-router.

  2. Record the CA and the client's leaf issuer:

    kubectl get secret -n podcertificate-controller-system pod-identity-ca-pool -o jsonpath='{.metadata.uid} {.metadata.creationTimestamp}'
kubectl exec -n test-client test-client -- sh -c 'openssl x509 -noout -issuer -enddate < /run/podidentity/credential-bundle.pem'

  3. Reinstall and recreate an actor:

    go run ./cmd/ate-setup delete ate-system
go run ./cmd/ate-setup deploy ate-system
go run ./cmd/ate-setup deploy demo counter
kubectl ate create actor s6-b -a ate-demo-counter --template counter

  4. Curl from the old pod and from a new pod with an identical spec:

    kubectl exec -n test-client test-client -- curl -sS -w "\nHTTP_STATUS=%{http_code}\n" \
  --cacert /run/servicedns-ca/trust-bundle.pem \
  --cert /run/podidentity/credential-bundle.pem \
  --key /run/podidentity/credential-bundle.pem \
  -H "ate-target-actor: ate-demo-counter/s6-b" \
  https://atenet-router.ate-system.svc/

  5. Kind only (static, not run on GKE):

    go run ./cmd/ate-setup deploy ate-system --kind
go run ./cmd/ate-setup delete ate-system --kind
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding | grep sandboxconfig-assets
Expected vs. Actual Behavior
  • Expected: Redeploying ate-system does not silently break mTLS for existing clients. Either the pod-identity CA is preserved across delete/deploy, the new trust bundle carries both CAs during a rollover, or the operator is told to recreate client pods. delete ate-system removes every cluster-scoped object that deploy ate-system created.
  • Actual: The old client pod fails with curl: (56) ... tlsv1 alert unknown ca, while a fresh pod with the same spec succeeds. The outage lasts until the old leaf is refreshed (inferred ~23h) or the pod is recreated. On --kind, the two sandboxconfig-assets admission objects are left behind (static).
Observed Output & Logs
  # pre-existing test-client (uid 0d2155a7):
  curl: (56) OpenSSL SSL_read: ... tlsv1 alert unknown ca
  # new test-client2 with identical spec:
  HTTP_STATUS=200
Lingua principale
Go
Stelle
6.1k
Fork
676
Merge medio
2g 41m
PR unite (30g)
500

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di agent-substrate/substrate

Tutte le issue di agent-substrate/substrate

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.