[Bug] Build id mismatch between controller and self-hosted server

オープン
  • 何か問題がありますか？
#693 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
go, kubernetes, prometheus
領域
backend, observability

調査の方向性

Start with the HPA generation path that injects the build ID and deployment name into metric selectors, then trace how those values are produced and compared with self-hosted Temporal server metrics. The report names examples/wrt-hpa-backlog.yaml as the reproduction; run it against a self-hosted server and inspect the generated HPA selectors alongside the backlog metric labels. Done means the selectors can match those labels and the backlog metric no longer fails, allowing scale-down.

索引モデルが issue の本文から書いたものです。

説明

bug
What are you really trying to do?

Scale each worker version on its task backlog with a self-hosted Temporal server, following Phase 2 of the HPA demo (examples/wrt-hpa-backlog.yaml). The demo uses temporal_cloud_v1_approximate_backlog_count; the self-hosted counterpart is approximate_backlog_count.

Describe the bug

The build ID the controller puts in each HPA's External metric selector can never match the build ID label on the server's backlog metrics.

The controller injects the raw build ID into the selector, and auto-generated build IDs always contain a - (<image tag>-<pod template hash>, see ComputeBuildID). The self-hosted server's default Prometheus reporter replaces every character outside [A-Za-z0-9_] in label values with _. For the same worker version:

Source Value
HPA selector (temporal_worker_build_id) service-e86a427173154a90-db48
Server metric (worker_build_id) service_e86a427173154a90_db48

The label names differ too (worker_build_id vs temporal_worker_build_id). A recording rule can rename those, but PromQL has no global string replace, so the values can't be mapped back.

Deployment names have the same problem whenever a WorkerDeployment's name contains - or .. The controller's label replaces only the / in <namespace>/<name>, while the server replaces all three characters.

As a result, the HPA reports <unknown> for the backlog metric with FailedGetExternalMetric. Since one of its metrics is failing, it also never scales down.

Minimal Reproduction

The demo's helloworld worker reproduces this without changes, run against a self-hosted server instead of Temporal Cloud:

  1. Install the Temporal Helm chart with the server's Prometheus metrics scraped (e.g. server.metrics.serviceMonitor.enabled: true).
  2. Deploy the helloworld WorkerDeployment and generate some load.
  3. Query the server's per-version backlog:
    group by (worker_deployment_name, worker_build_id) (approximate_backlog_count{worker_deployment_name!=""})
  4. Apply a WorkerResourceTemplate with an External metric on that series (renamed by a recording rule if you like), then compare the injected selector:
    kubectl get hpa -o jsonpath='{range .items[*]}{.spec.metrics[*].external.metric.selector.matchLabels}{"\n"}{end}'

The build IDs differ only in - vs _.

Environment/Versions
  • OS and processor: Linux x86_64
  • temporal-worker-controller v1.14.0 (chart 0.33.0)
  • Temporal Server 1.32.2 (Helm chart 1.8.0)
  • Kubernetes v1.37.0 on kind, with kube-prometheus-stack and prometheus-adapter
Additional context

My agent came up with this workaround: allow - and . in label values through the server's Prometheus sanitizeOptions. After that, the server's labels match the controller's values exactly:

global:
  metrics:
    prometheus:
      sanitizeOptions:
        nameChars: &alphanumericUnderscore
          ranges: &alphanumeric
            - { startRange: "a", endRange: "z" }
            - { startRange: "A", endRange: "Z" }
            - { startRange: "0", endRange: "9" }
          safeChars: "_"
        keyChars: *alphanumericUnderscore
        valueChars:
          ranges: *alphanumeric
          safeChars: "_-."
        replacementChar: "_"

But sanitizeOptions is marked deprecated and it changes label values on every server metric, so that doesn't seem great.

主要言語
Go
スター
199
フォーク
85
平均マージ
2日 15時間
マージ済み PR（30日）
33

環境構築

  • Dockerfile または Docker Compose ファイルあり
  • プルリクエストのテンプレートなし
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

temporalio/temporal-worker-controller のほかの issue

temporalio/temporal-worker-controller の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。