helm/helm

get-helm-N installs inconsistent versions due to CDN caching

オープン

#32,329 opened on 2026/07/10

 (2 件のコメント) (0 件のリアクション) (0 人の担当者)Go (7,602 件のフォーク)batch import
bughelp wantedkeep open

Repository metrics

Stars
 (29,815 個のスター)
PR merge metrics
 (平均マージ 40d 17h) (30d で 48 merged PRs)

説明

Summary

get_helm-3 queries https://get.helm.sh/helm3-latest-version to determine the latest version, but the request has no cache-control headers or cache-busting query string. CDN edge servers can serve stale cached responses for hours after a new Helm release. This means get_helm-3 running on different machines at the same time can install different versions depending on which CDN edge server they reach.

Context

We discovered this while debugging a flaky integration test in minikube: kubernetes/minikube#23323

Helm v3.21.3 was released on Jul 9, 2026 ~22:00 UTC. About 19 hours later, our CI ran get_helm.sh on multiple machines:

  • Some machines installed v3.21.3 (hit updated CDN edge nodes)
  • Some machines installed v3.21.2 (hit stale CDN edge nodes)
  • One machine installed v3.21.2 on the first run and v3.21.3 on the second run minutes later

Impact

After a new Helm release, get_helm-3 becomes non-deterministic. Running it on different machines (or even the same machine at different times) can install different versions. This makes it an "install random version from {current-1, current}" tool rather than a reliable "install latest" tool.

This affects:

  • CI/CD pipelines that run get_helm.sh on multiple nodes and expect consistent versions
  • Multi-machine deployments where all machines should have the same Helm version
  • Any automation that compares the installed version to an expected "latest"

Current cache behavior

The version file is served from Azure Blob Storage behind Azure Front Door CDN. The origin sets no Cache-Control header:

$ curl -sI 'https://get.helm.sh/helm3-latest-version'
HTTP/2 200 
date: Fri, 10 Jul 2026 18:45:44 GMT
content-type: application/octet-stream
content-length: 8
content-md5: CsRKs3WCs5c4CaZod7y6XQ==
last-modified: Thu, 09 Jul 2026 21:12:31 GMT
etag: 0x8DEDDFECA26742A
x-ms-request-id: f772980b-101e-000f-169c-108358000000
x-ms-version: 2009-09-19
x-ms-lease-status: unlocked
x-ms-blob-type: BlockBlob
x-azure-ref: 20260710T184544Z-16cf95fd5c9dfrdjhC1FRAss740000001uhg000000001dap
x-fd-int-roxy-purgeid: 4
x-cache-info: L2_T2
x-cache: TCP_REMOTE_HIT
accept-ranges: bytes

Without an explicit Cache-Control header, Azure Front Door applies its own default TTL, which can be hours. The x-cache: TCP_REMOTE_HIT and x-cache-info: L2_T2 headers confirm the response is served from a multi-tier CDN cache, not the origin.

Setting Cache-Control: max-age=60 on the helm3-latest-version blob in Azure Storage would ensure all CDN edge nodes refresh within 60 seconds of a release.

Suggested fixes

CDN configuration

Configure the CDN to use a short TTL or active cache purge for the version file (helm3-latest-version). The CDN can:

  • Set a short TTL (e.g. 60 seconds) on the version file
  • Actively purge the version file from all edge nodes when a release is published (most CDNs have purge APIs)
  • Return Cache-Control: no-cache or max-age=60 on the version file response, so downstream caches also keep short TTLs

The version file is ~10 bytes and queried only during install/upgrade — a short TTL has negligible cost. The binary tarballs (which are large) should stay cached with long TTLs — only the version file needs fast propagation.

Client-side cache-busting in get-helm-3

Add cache-busting to the curl/wget request in checkDesiredVersion():

# Current:
latest_release_response=$( curl -L --silent --show-error --fail "$latest_release_url" 2>&1 || true )

# Suggested:
latest_release_response=$( curl -L --silent --show-error --fail \
  -H "Cache-Control: no-cache" \
  "$latest_release_url?t=$(date +%s)" 2>&1 || true )
  • Cache-Control: no-cache asks well-behaved intermediaries to revalidate
  • ?t=$(date +%s) forces a cache miss on CDNs that treat each unique URL as a separate cache key

This cannot guarantee bypassing all intermediate caches (ISP proxies, corporate proxies, etc.), but it handles most cases. The same change should be applied to the wget fallback path.

Issues:

  • Will cause higher load on the origin server since CDN caching is bypassed; the response is ~10 bytes, so the additional load should be negligible

コントリビューターガイド