[Bug] Build id mismatch between controller and self-hosted server

未关闭
  • 哪里不对？
#693 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
go, kubernetes, prometheus
领域
backend, observability

调研方向

Start with the HPA generation path that injects the build ID and deployment name into metric selectors, then trace how those values are produced and compared with self-hosted Temporal server metrics. The report names examples/wrt-hpa-backlog.yaml as the reproduction; run it against a self-hosted server and inspect the generated HPA selectors alongside the backlog metric labels. Done means the selectors can match those labels and the backlog metric no longer fails, allowing scale-down.

由索引模型根据 Issue 内容生成。

描述

bug
What are you really trying to do?

Scale each worker version on its task backlog with a self-hosted Temporal server, following Phase 2 of the HPA demo (examples/wrt-hpa-backlog.yaml). The demo uses temporal_cloud_v1_approximate_backlog_count; the self-hosted counterpart is approximate_backlog_count.

Describe the bug

The build ID the controller puts in each HPA's External metric selector can never match the build ID label on the server's backlog metrics.

The controller injects the raw build ID into the selector, and auto-generated build IDs always contain a - (<image tag>-<pod template hash>, see ComputeBuildID). The self-hosted server's default Prometheus reporter replaces every character outside [A-Za-z0-9_] in label values with _. For the same worker version:

Source Value
HPA selector (temporal_worker_build_id) service-e86a427173154a90-db48
Server metric (worker_build_id) service_e86a427173154a90_db48

The label names differ too (worker_build_id vs temporal_worker_build_id). A recording rule can rename those, but PromQL has no global string replace, so the values can't be mapped back.

Deployment names have the same problem whenever a WorkerDeployment's name contains - or .. The controller's label replaces only the / in <namespace>/<name>, while the server replaces all three characters.

As a result, the HPA reports <unknown> for the backlog metric with FailedGetExternalMetric. Since one of its metrics is failing, it also never scales down.

Minimal Reproduction

The demo's helloworld worker reproduces this without changes, run against a self-hosted server instead of Temporal Cloud:

  1. Install the Temporal Helm chart with the server's Prometheus metrics scraped (e.g. server.metrics.serviceMonitor.enabled: true).
  2. Deploy the helloworld WorkerDeployment and generate some load.
  3. Query the server's per-version backlog:
    group by (worker_deployment_name, worker_build_id) (approximate_backlog_count{worker_deployment_name!=""})
  4. Apply a WorkerResourceTemplate with an External metric on that series (renamed by a recording rule if you like), then compare the injected selector:
    kubectl get hpa -o jsonpath='{range .items[*]}{.spec.metrics[*].external.metric.selector.matchLabels}{"\n"}{end}'

The build IDs differ only in - vs _.

Environment/Versions
  • OS and processor: Linux x86_64
  • temporal-worker-controller v1.14.0 (chart 0.33.0)
  • Temporal Server 1.32.2 (Helm chart 1.8.0)
  • Kubernetes v1.37.0 on kind, with kube-prometheus-stack and prometheus-adapter
Additional context

My agent came up with this workaround: allow - and . in label values through the server's Prometheus sanitizeOptions. After that, the server's labels match the controller's values exactly:

global:
  metrics:
    prometheus:
      sanitizeOptions:
        nameChars: &alphanumericUnderscore
          ranges: &alphanumeric
            - { startRange: "a", endRange: "z" }
            - { startRange: "A", endRange: "Z" }
            - { startRange: "0", endRange: "9" }
          safeChars: "_"
        keyChars: *alphanumericUnderscore
        valueChars:
          ranges: *alphanumeric
          safeChars: "_-."
        replacementChar: "_"

But sanitizeOptions is marked deprecated and it changes label values on every server metric, so that doesn't seem great.

主要语言
Go
星标
199
派生
85
平均合并
2 天 15 小时
30 天内合并 PR
33

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

temporalio/temporal-worker-controller 的其他 Issue

查看 temporalio/temporal-worker-controller 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。