Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

k8s-health-check example does not work if using ASM in GKE

Aperta
  • Qualcosa non va?
#2,706 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
gcp, kubernetes
Ambito
cloud, devops

Direzione di ricerca

The failing setup is the native-sidecar example in examples/k8s-health-check/proxy_with_http_health_check.yaml, where the cloud-sql-proxy initContainer's startupProbe targets the named port http-sqlproxy (9090). The issue hinges on how the managed service mesh rewrites that probe port, so the first step is reproducing the failure on a GKE cluster with ASM enabled. Done would mean a documented or tested configuration in which the proxy's startup probe succeeds under ASM, which likely requires a maintainer decision on the supported approach.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

type: bug
Bug Description

Background

I have an application where the CloudSQL Auth Proxy must be up and working at startup time (it makes a SQL query right at startup).

To ensure this works well, I have incorporated the k8s-health-check example from here. This runs the proxy as a native sidecar together with a startup probe, which ensures the service is up and running before my application starts.

This has been working fine for some time, but recently we chose to adopt ASM in GKE for service mesh purposes. This has broken this existing approach.

Problem

We have encountered a circular dependency between the CloudSQL Auth Proxy's startup probe port and istio starting, which is caused because the auth proxy runs as a native sidecar but istio runs as a legacy sidecar...

Istio cannot start until all the initContainers have completed (aka: native sidecars are up and running), but the CloudSQL Auth Proxy's startup probe will never succeed because the service mesh has re-written the startup probe's port number to route through istio, which isn't running yet.

To be specific:

  • CloudSQL Auth Proxy was defined as a native sidecar (aka: an initContainer with restart: Always), exactly as documented in the installation examples here.

  • The istio service mesh sidecar was started as a legacy sidecar, which starts after the native sidecars.

  • ASM cannot be configured to use a native sidecar, see Supported features using Istio APIs (managed control plane) docs:

Image

Workaround (deprecated)

I have a workaround, which was to set these annotations on my deployment:

        traffic.sidecar.istio.io/excludeInboundPorts: "9090"
        sidecar.istio.io/rewriteAppHTTPProbers: false

(note: port 9090 is the http-port defined on the CloudSQL proxy)

This prevented the service mesh from rewriting the startup probe's port on the container, which in turn meant the probes would succeed and the application would work.

However, this workaround has just been deprecated by Google. Specifically, the idecar.istio.io/rewriteAppHTTPProbers: false annotation is part of istiod, which Google have deprecated in favour of Traffic Director, and this doesn't support this annotation.

Therefore, my workaround is not sufficient as we need to migrate from istiod to Traffic Director.

Please can you assist in a suggested fix for this?

Example code (or command)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portal
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: portal
    app.kubernetes.io/component: backend
spec:
  progressDeadlineSeconds: 360
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: portal
  template:
    metadata:
      labels:
        app: portal
        app.kubernetes.io/name: portal
        app.kubernetes.io/component: backend
        istio.io/rev: asm-managed-rapid
      annotations:
        prometheus.io/scrape: 'true'
        # Service metrics
        prometheus.io/port: http-port
        prometheus.io/path: "/metrics"
        # Cloud SQL Auth Proxy metrics
        prometheus.io/port2: http-sqlproxy
        prometheus.io/path2: "/metrics"
        traffic.sidecar.istio.io/excludeInboundPorts: "9090"
        sidecar.istio.io/rewriteAppHTTPProbers: false
    spec:
      serviceAccountName: portal
      restartPolicy: Always
      initContainers:
        - name: cloud-sql-proxy
          restartPolicy: Always
          image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.25.3
          command:
            - "/cloud-sql-proxy"
            - '--structured-logs'
            - '--psc'
            - '--auto-iam-authn'
            # Serve health checks & prometheus metrics
            - '--health-check'
            - '--prometheus'
            - "--http-address=0.0.0.0"
            - "--http-port=9090"
            - "{{ .Values.database.instanceConnectionName }}?port=3306"
          ports:
            - name: http-sqlproxy
              containerPort: 9090
              protocol: TCP
          startupProbe:
            httpGet:
              path: /startup
              port: http-sqlproxy
            periodSeconds: 1
            timeoutSeconds: 5
            failureThreshold: 20
          livenessProbe:
            httpGet:
              path: /liveness
              port: http-sqlproxy
            initialDelaySeconds: 0
            periodSeconds: 60
            timeoutSeconds: 30
            failureThreshold: 5
          resources:
            requests:
              memory: {{ .Values.resources.cloudSqlProxy.memory }}
              cpu: {{ .Values.resources.cloudSqlProxy.cpu }}
            limits:
              memory: {{ .Values.resources.cloudSqlProxy.memory }}
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
      containers:
        - name: portal
          image: {{ .Values.image }}
          ports:
            - name: http-port
              containerPort: 8000
          startupProbe:
            httpGet:
              path: /ready
              port: http-port
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 1
            timeoutSeconds: 1
            failureThreshold: 30  # Allow up to 30 seconds for startup, checking every 1s
          readinessProbe:
            httpGet:
              path: /ready
              port: http-port
              scheme: HTTP
          livenessProbe:
            httpGet:
              path: /health
              port: http-port
...
Steps to reproduce?
  1. Incorportae k8s-health-check example from here
  2. Enable ASM on your cluster
  3. Your service will no longer start, as the SQL proxy no longer starts
    ...
Environment
  1. Cloud SQL Proxy version (./cloud-sql-proxy --version): 2.25.3
  2. Proxy invocation command (for example, ./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME): See above 'code' example.
Lingua principale
Go
Stelle
1.4k
Fork
357
Merge medio
1g 18h
PR unite (30g)
18

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di GoogleCloudPlatform/cloud-sql-proxy

Tutte le issue di GoogleCloudPlatform/cloud-sql-proxy

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.