k8s-health-check example does not work if using ASM in GKE

The failing setup is the native-sidecar example in examples/k8s-health-check/proxy_with_http_health_check.yaml, where the cloud-sql-proxy initContainer's startupProbe targets the named port http-sqlproxy (9090). The issue hinges on how the managed service mesh rewrites that probe port, so the first step is reproducing the failure on a GKE cluster with ASM enabled. Done would mean a documented or tested configuration in which the proxy's startup probe succeeds under ASM, which likely requires a maintainer decision on the supported approach.

Bug Description

Background

I have an application where the CloudSQL Auth Proxy must be up and working at startup time (it makes a SQL query right at startup).

To ensure this works well, I have incorporated the k8s-health-check example from here. This runs the proxy as a native sidecar together with a startup probe, which ensures the service is up and running before my application starts.

This has been working fine for some time, but recently we chose to adopt ASM in GKE for service mesh purposes. This has broken this existing approach.

Problem

We have encountered a circular dependency between the CloudSQL Auth Proxy's startup probe port and istio starting, which is caused because the auth proxy runs as a native sidecar but istio runs as a legacy sidecar...

Istio cannot start until all the initContainers have completed (aka: native sidecars are up and running), but the CloudSQL Auth Proxy's startup probe will never succeed because the service mesh has re-written the startup probe's port number to route through istio, which isn't running yet.

To be specific:

• CloudSQL Auth Proxy was defined as a native sidecar (aka: an initContainer with restart: Always), exactly as documented in the installation examples here.

• The istio service mesh sidecar was started as a legacy sidecar, which starts after the native sidecars.

• ASM cannot be configured to use a native sidecar, see Supported features using Istio APIs (managed control plane) docs:

Workaround (deprecated)

I have a workaround, which was to set these annotations on my deployment:

        traffic.sidecar.istio.io/excludeInboundPorts: "9090"
        sidecar.istio.io/rewriteAppHTTPProbers: false

(note: port 9090 is the http-port defined on the CloudSQL proxy)

This prevented the service mesh from rewriting the startup probe's port on the container, which in turn meant the probes would succeed and the application would work.

However, this workaround has just been deprecated by Google. Specifically, the idecar.istio.io/rewriteAppHTTPProbers: false annotation is part of istiod, which Google have deprecated in favour of Traffic Director, and this doesn't support this annotation.

Therefore, my workaround is not sufficient as we need to migrate from istiod to Traffic Director.

Please can you assist in a suggested fix for this?

Example code (or command)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: portal
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: portal
    app.kubernetes.io/component: backend
spec:
  progressDeadlineSeconds: 360
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: portal
  template:
    metadata:
      labels:
        app: portal
        app.kubernetes.io/name: portal
        app.kubernetes.io/component: backend
        istio.io/rev: asm-managed-rapid
      annotations:
        prometheus.io/scrape: 'true'
        # Service metrics
        prometheus.io/port: http-port
        prometheus.io/path: "/metrics"
        # Cloud SQL Auth Proxy metrics
        prometheus.io/port2: http-sqlproxy
        prometheus.io/path2: "/metrics"
        traffic.sidecar.istio.io/excludeInboundPorts: "9090"
        sidecar.istio.io/rewriteAppHTTPProbers: false
    spec:
      serviceAccountName: portal
      restartPolicy: Always
      initContainers:
        - name: cloud-sql-proxy
          restartPolicy: Always
          image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.25.3
          command:
            - "/cloud-sql-proxy"
            - '--structured-logs'
            - '--psc'
            - '--auto-iam-authn'
            # Serve health checks & prometheus metrics
            - '--health-check'
            - '--prometheus'
            - "--http-address=0.0.0.0"
            - "--http-port=9090"
            - "{{ .Values.database.instanceConnectionName }}?port=3306"
          ports:
            - name: http-sqlproxy
              containerPort: 9090
              protocol: TCP
          startupProbe:
            httpGet:
              path: /startup
              port: http-sqlproxy
            periodSeconds: 1
            timeoutSeconds: 5
            failureThreshold: 20
          livenessProbe:
            httpGet:
              path: /liveness
              port: http-sqlproxy
            initialDelaySeconds: 0
            periodSeconds: 60
            timeoutSeconds: 30
            failureThreshold: 5
          resources:
            requests:
              memory: {{ .Values.resources.cloudSqlProxy.memory }}
              cpu: {{ .Values.resources.cloudSqlProxy.cpu }}
            limits:
              memory: {{ .Values.resources.cloudSqlProxy.memory }}
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
      containers:
        - name: portal
          image: {{ .Values.image }}
          ports:
            - name: http-port
              containerPort: 8000
          startupProbe:
            httpGet:
              path: /ready
              port: http-port
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 1
            timeoutSeconds: 1
            failureThreshold: 30  # Allow up to 30 seconds for startup, checking every 1s
          readinessProbe:
            httpGet:
              path: /ready
              port: http-port
              scheme: HTTP
          livenessProbe:
            httpGet:
              path: /health
              port: http-port
...

Steps to reproduce?

1. Incorportae k8s-health-check example from here
2. Enable ASM on your cluster
3. Your service will no longer start, as the SQL proxy no longer starts
   ...

Environment

1. Cloud SQL Proxy version (./cloud-sql-proxy --version): 2.25.3
2. Proxy invocation command (for example, ./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME): See above 'code' example.