Dot in HTTPRoute name produces duplicate envoy_cluster_total_match_count Prometheus series

Description:

When an HTTPRoute's name contains a dot (a valid Kubernetes name, e.g. echo.a.example.com), the Envoy proxy emits envoy_cluster_total_match_count on /stats/prometheus with a truncated envoy_cluster_name label (cut at the first dot). When two clusters share the same before-the-first-dot prefix — two HTTPRoutes whose names differ only after a dot, or one dotted HTTPRoute with several rules — the same series (identical metric name and labels) is emitted once per cluster in a single scrape response:

envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 6
envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1

Once the two values diverge (they are distinct counters), Prometheus rejects the samples — "Error on ingesting samples with different value but same timestamp" — incrementing prometheus_target_scrapes_sample_duplicate_timestamp_total and firing the kube-prometheus-stack PrometheusDuplicateTimestamps alert.

Expected: each {__name__, labels} series appears at most once per response, and envoy_cluster_name carries the full cluster name.

See "Possible root cause" at the bottom.

Repro steps:

Stock Envoy Gateway v1.8.2 on kind — no EnvoyProxy customization:

kind create cluster --name eg-dup
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.8.2 \
  -n envoy-gateway-system --create-namespace
kubectl wait --timeout=300s -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
kubectl apply -f repro.yaml   # manifests below
kubectl wait --timeout=120s -n default deployment/echo --for=condition=Available

POD=$(kubectl -n envoy-gateway-system get pods -l gateway.envoyproxy.io/owning-gateway-name=eg -o name | head -1)
kubectl -n envoy-gateway-system port-forward "$POD" 19001:19001 19000:19000 &
sleep 2

Output (deterministic, no traffic needed for the duplication itself). The two clusters hold two distinct counters:

curl -s 'localhost:19000/stats?filter=total_match_count' | grep httproute

cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count: 1
cluster.httproute/default/echo.b.example.com/rule/0.default.total_match_count: 1

but on /stats/prometheus both render as the same series, emitted twice:

curl -s localhost:19001/stats/prometheus | grep 'envoy_cluster_total_match_count{' | grep httproute

envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1
envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1

Being distinct counters, their values diverge as soon as the clusters open different numbers of upstream connections — at which point Prometheus starts rejecting the samples (see Logs).

repro.yaml — one Gateway, one echo backend, and two HTTPRoutes whose names contain dots and share the prefix before the first dot:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: {app: echo}
  template:
    metadata:
      labels: {app: echo}
    spec:
      containers:
        - name: echo
          image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          ports: [{containerPort: 3000}]
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: default
spec:
  selector: {app: echo}
  ports: [{port: 3000, targetPort: 3000}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo.a.example.com
  namespace: default
spec:
  parentRefs: [{name: eg}]
  hostnames: ["echo-a.example.com"]
  rules:
    - backendRefs: [{name: echo, port: 3000}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo.b.example.com
  namespace: default
spec:
  parentRefs: [{name: eg}]
  hostnames: ["echo-b.example.com"]
  rules:
    - backendRefs: [{name: echo, port: 3000}]

We originally hit this in production: our controller names HTTPRoutes after service hostnames (dots included), and every affected proxy showed the duplicated series for each such cluster, with intermittent Prometheus scrape rejections under traffic.

Environment:

• Envoy Gateway v1.8.2
• Envoy 0ebfcfe5b0484b89ca85b761da9e05ce75dbda8d/1.38.3/Clean/RELEASE/BoringSSL
• Kubernetes: kind (repro); EKS 1.35 (production)
• Prometheus via kube-prometheus-stack, scraping /stats/prometheus on :19001

Logs:

Prometheus scrape rejection when the duplicate copies diverge under traffic (production):

level=WARN source=scrape_append_v2.go:402 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/envoy-gateway-system/envoy-gateway-proxy/0 target=http://<proxy-ip>:19001/stats/prometheus num_dropped=4

Possible root cause:

The bootstrap template (internal/xds/bootstrap/bootstrap.yaml.tpl) ships this stats tag:

- regex: "^cluster(\\..+\\.(.+))\\.total_match_count$"
  tag_name: socket_match_name

For a raw stat cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count, this greedy regex removes the entire middle (.httproute/default/echo.a.example.com/rule/0.default) from the tag-extracted name, while Envoy's default envoy_cluster_name extractor takes the cluster name only up to the first dot (httproute/default/echo). Everything after the first dot — the part that distinguishes the clusters — is discarded from both the name and the labels, so distinct clusters produce identical series:

# raw admin /stats — two distinct counters:
cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count: 6
cluster.httproute/default/echo.b.example.com/rule/0.default.total_match_count: 1

Envoy Gateway puts the user-controlled HTTPRoute name verbatim into the cluster name (httproute/<ns>/<name>/rule/<n>), so any dotted route name triggers this. Only total_match_count collides into exact duplicates (it is the only cluster stat whose tag regex strips the whole middle); other stats of dotted routes instead leak the post-dot remainder into the metric name (e.g. envoy_cluster_a.example.com/rule/0_upstream_rq_total).

Prepared with the assistance of an AI agent; the reporter has reviewed and understands the contents.