grafana/mimir

[helm] Ingress routes to non-existent mimir-alertmanager service when zoneAwareReplication is enabled

Open

#14.334 geöffnet am 12. Feb. 2026

Auf GitHub ansehen
 (3 Kommentare) (5 Reaktionen) (0 zugewiesene Personen)Go (808 Forks)auto 404
bughelmhelp wanted

Repository-Metriken

Stars
 (5.181 Stars)
PR-Merge-Metriken
 (PR-Metriken ausstehend)

Beschreibung

What is the bug?

After upgrading from mimir-distributed-5.8.0 to mimir-distributed-6.0.5, the Ingress for alertmanager paths breaks with:

<error: services "mimir-alertmanager" not found>

This affects all alertmanager ingress paths:

  • /alertmanager
  • /multitenant_alertmanager/status
  • /multitenant_alertmanager/configs
  • /api/v1/alerts

Root Cause

PR #13064 introduced this breaking change in 6.0.0:

[CHANGE] Ingress: Change default routes to point to non-headless services. #13064

The ingress template was changed to use mimir-alertmanager (ClusterIP) instead of mimir-alertmanager-headless. However, when alertmanager.zoneAwareReplication.enabled=true, the chart never creates a mimir-alertmanager ClusterIP service — only zone-specific services and the headless service are created.

Services that exist with zoneAwareReplication enabled:

mimir-alertmanager-headless   ClusterIP   None    8080/TCP 
mimir-alertmanager-zone-a     ClusterIP   <IP>   8080/TCP 
mimir-alertmanager-zone-b     ClusterIP   <IP>    8080/TCP 
mimir-alertmanager-zone-c     ClusterIP   <IP>   8080/TCP 

Steps to Reproduce

  1. Install mimir-distributed chart with alertmanager.zoneAwareReplication.enabled: true
  2. Enable ingress.enabled: true
  3. Upgrade from any 5.x chart to 6.0.x
  4. Run kubectl describe ingress mimir -n <namespace>
  5. Observe <error: services "mimir-alertmanager" not found> on alertmanager paths

Expected Behavior

Ingress should route alertmanager paths to mimir-alertmanager-headless when zoneAwareReplication.enabled=true, since the non-headless service is not created in that configuration.

Actual Behavior

Ingress tries to route to mimir-alertmanager which does not exist, causing all alertmanager paths to return errors.

Workaround

Override alertmanager ingress paths in values.yaml to explicitly use the headless service:

ingress:
  paths:
    alertmanager:
    alertmanager-headless:
      - path: /alertmanager
      - path: /multitenant_alertmanager/status
      - path: /multitenant_alertmanager/configs
      - path: /api/v1/alerts

Suggested Fix

The ingress template should conditionally use the headless service for alertmanager when zoneAwareReplication.enabled=true:

{{- if .Values.alertmanager.zoneAwareReplication.enabled }}
name: mimir-alertmanager-headless
{{- else }}
name: mimir-alertmanager
{{- end }}

Environment

Chart version mimir-distributed-6.0.5
Mimir version 3.0.1
Kubernetes AKS (Azure)
Ingress controller nginx
zoneAwareReplication enabled (3 zones)

Helm values (relevant section)

alertmanager:
  zoneAwareReplication:
    enabled: true
    zones:
      - name: zone-a
        ...
      - name: zone-b
        ...
      - name: zone-c
        ...

ingress:
  enabled: true
  hosts:
    - mimir.example.com

Relevant Links

Contributor Guide