[Bug] Build id mismatch between controller and self-hosted server

Abierto
  • ¿Algo no está bien?
#693 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go, kubernetes, prometheus
Área
backend, observability

Línea de trabajo

Start with the HPA generation path that injects the build ID and deployment name into metric selectors, then trace how those values are produced and compared with self-hosted Temporal server metrics. The report names examples/wrt-hpa-backlog.yaml as the reproduction; run it against a self-hosted server and inspect the generated HPA selectors alongside the backlog metric labels. Done means the selectors can match those labels and the backlog metric no longer fails, allowing scale-down.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
What are you really trying to do?

Scale each worker version on its task backlog with a self-hosted Temporal server, following Phase 2 of the HPA demo (examples/wrt-hpa-backlog.yaml). The demo uses temporal_cloud_v1_approximate_backlog_count; the self-hosted counterpart is approximate_backlog_count.

Describe the bug

The build ID the controller puts in each HPA's External metric selector can never match the build ID label on the server's backlog metrics.

The controller injects the raw build ID into the selector, and auto-generated build IDs always contain a - (<image tag>-<pod template hash>, see ComputeBuildID). The self-hosted server's default Prometheus reporter replaces every character outside [A-Za-z0-9_] in label values with _. For the same worker version:

Source Value
HPA selector (temporal_worker_build_id) service-e86a427173154a90-db48
Server metric (worker_build_id) service_e86a427173154a90_db48

The label names differ too (worker_build_id vs temporal_worker_build_id). A recording rule can rename those, but PromQL has no global string replace, so the values can't be mapped back.

Deployment names have the same problem whenever a WorkerDeployment's name contains - or .. The controller's label replaces only the / in <namespace>/<name>, while the server replaces all three characters.

As a result, the HPA reports <unknown> for the backlog metric with FailedGetExternalMetric. Since one of its metrics is failing, it also never scales down.

Minimal Reproduction

The demo's helloworld worker reproduces this without changes, run against a self-hosted server instead of Temporal Cloud:

  1. Install the Temporal Helm chart with the server's Prometheus metrics scraped (e.g. server.metrics.serviceMonitor.enabled: true).
  2. Deploy the helloworld WorkerDeployment and generate some load.
  3. Query the server's per-version backlog:
    group by (worker_deployment_name, worker_build_id) (approximate_backlog_count{worker_deployment_name!=""})
  4. Apply a WorkerResourceTemplate with an External metric on that series (renamed by a recording rule if you like), then compare the injected selector:
    kubectl get hpa -o jsonpath='{range .items[*]}{.spec.metrics[*].external.metric.selector.matchLabels}{"\n"}{end}'

The build IDs differ only in - vs _.

Environment/Versions
  • OS and processor: Linux x86_64
  • temporal-worker-controller v1.14.0 (chart 0.33.0)
  • Temporal Server 1.32.2 (Helm chart 1.8.0)
  • Kubernetes v1.37.0 on kind, with kube-prometheus-stack and prometheus-adapter
Additional context

My agent came up with this workaround: allow - and . in label values through the server's Prometheus sanitizeOptions. After that, the server's labels match the controller's values exactly:

global:
  metrics:
    prometheus:
      sanitizeOptions:
        nameChars: &alphanumericUnderscore
          ranges: &alphanumeric
            - { startRange: "a", endRange: "z" }
            - { startRange: "A", endRange: "Z" }
            - { startRange: "0", endRange: "9" }
          safeChars: "_"
        keyChars: *alphanumericUnderscore
        valueChars:
          ranges: *alphanumeric
          safeChars: "_-."
        replacementChar: "_"

But sanitizeOptions is marked deprecated and it changes label values on every server metric, so that doesn't seem great.

Lenguaje dominante
Go
Estrellas
199
Forks
85
Merge medio
2 d 15 h
PR fusionados (30 d)
33

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de temporalio/temporal-worker-controller

Todos los issues de temporalio/temporal-worker-controller

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.