Filter sidebar ServiceName values are incomplete — unordered `LIMIT` sample misses services as ingest volume grows
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 52/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- clickhouse, react, typescript
- 領域
- frontend, observability, search
調査の方向性
Start by tracing the Search page sidebar filter request to the code that builds the ClickHouse query; the issue identifies /api/clickhouse-proxy and shows the current SQL pattern. Check the filter's query and result handling, then verify that distinct values beyond the first 3,000,000 input rows can appear and whether typing triggers a server request. Done means the sidebar can find known values from the selected time range without silently hiding them.
索引モデルが issue の本文から書いたものです。
説明
Update: root cause identified. See "Actual query and root cause" below — this replaces the original "no network request fires" theory, which turned out to be a symptom, not the cause.
Description
The Search page's filter sidebar for ServiceName (and presumably any other top-level column) does not reliably surface all distinct values. At high ingest volume, specific known-good values never appear in the filter list or its search box, no matter how they're typed.
This used to work correctly on an older HyperDX version (roughly 3-4 months ago, ClickHouse-side ingest volume was significantly lower at the time). It broke gradually as traffic grew — not because of a version regression, but because of how the underlying query is structured.
Actual query and root cause
Captured from the browser (/api/clickhouse-proxy, fired once when the ServiceName filter is opened):
WITH sampledData AS (
SELECT ServiceName AS param0
FROM default.otel_traces
WHERE Timestamp >= fromUnixTimestamp64Milli(<start>)
AND Timestamp <= fromUnixTimestamp64Milli(<end>)
LIMIT 3000000
)
SELECT groupUniqArray(10000)(param0) AS param0
FROM sampledData
This does not compute SELECT DISTINCT ServiceName (or an aggregated GROUP BY) over the full time range. It takes an unordered LIMIT 3,000,000 slice of raw rows from the time range, then extracts up to 10,000 distinct ServiceName values from that slice only. Any ServiceName not present in the first 3,000,000 rows the scan happens to read is invisible to the filter — permanently, since no further request fires when the user types in the filter's search box (confirmed via Network tab: zero new requests on typing, only client-side filtering of the one cached result).
Why this is volume-dependent
The otel_traces table (standard ClickStack/OTel schema) is defined as:
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
ServiceName is the first sort key, so an unordered scan reads rows in roughly that physical order. In our environment, one single ServiceName alone produced 52 million rows in a 16-minute window — more than 17x the entire 3,000,000-row sample budget by itself. Total ingest across all services in that same window is well over 100M+ rows. A fixed 3,000,000-row sample therefore covers under ~2% of the window, and gets exhausted by whichever services are read "early" (by part/thread order) long before the scan reaches services that sort later or simply live in parts read later — regardless of how important or high-volume those services actually are.
Three to four months ago, with far lower ingest volume, the same fixed 3,000,000-row budget would have covered the large majority (or all) of the window's rows, so effectively every ServiceName showed up — hence "it worked like butter" on an older deployment. As traffic grew, the same code path started silently dropping services with no error, no truncation warning, and no way to recover them via the search box (since search doesn't issue a new query).
Environment
- HyperDX version: 2.42.0 (self-hosted via ClickStack docker-compose)
- Source: OTel traces (
otel_traces),ORDER BY (ServiceName, SpanName, Timestamp), hourly partitions ServiceNamecardinality: ~1,300 distinct values in a 16-minute window (consumer/job-per-feature naming, not a bounded microservice list)- Row volume: 52M+ rows for a single
ServiceNamein a 16-minute window; 100M+ total across all services in the same window
Steps to reproduce
- Have a table where one or a few
ServiceName/SpanNamevalues dominate row volume (common for high-traffic OTel traces tables). - Open the Search page's
ServiceNamefilter in the sidebar. - Search for a
ServiceNameknown to exist in ClickHouse for the selected time range, but with lower read-order priority relative to the dominant service(s). - Observe it never appears, and typing in the search box fires no new network request.
Expected
The filter should surface all distinct values that actually exist in the selected time range, or at minimum should sample representatively (e.g., via GROUP BY/DISTINCT with a result-count limit, not an input-row-count limit), and/or should issue a fresh, filtered query when the user searches instead of silently filtering a possibly-incomplete cached list.
Actual
Deterministically misses ServiceName values once per-window row volume for other services exceeds the fixed 3,000,000-row sample budget. No error or truncation indicator is shown to the user. Typing to search never re-queries the server.
Suggested fix
Replace the LIMIT <input rows> + groupUniqArray pattern with a query whose limit is on the output (distinct values), not the input (raw rows read) — e.g. SELECT ServiceName FROM otel_traces WHERE <time range> GROUP BY ServiceName LIMIT 10000, or SELECT DISTINCT ServiceName ... LIMIT 10000. This scans the full time range and returns correct results up to the 10k value cap, instead of an arbitrary, volume-dependent, order-dependent slice. If the row-limited sampling exists specifically to bound query cost on very large tables, consider sampling proportionally/randomly (e.g. SAMPLE clause) rather than taking the physically-first N rows, and/or surfacing a "results may be incomplete" indicator when the input-row cap is hit.
Workaround
Bypass the sidebar facet entirely and filter directly via the query bar (ServiceName:"exact-value" in Lucene mode, or ServiceName = 'exact-value' in SQL mode) — this goes straight to ClickHouse and isn't affected by the sampling issue.
- 主要言語
- TypeScript
- スター
- 10.1k
- フォーク
- 527
- 平均マージ
- 1日 20時間
- マージ済み PR(30日)
- 186
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
ClickHouse/ClickStack のほかの issue
-
external
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
ClickHouse/ClickStack#3761 ·
メンテナーはふだん 1 日以内に返信
-
Metric chart with aggFn increase and a multi-column group-by fails: toString() gets several arguments対応中かも @CG-5228 が 26 日前に担当しました。 オープンexternal
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
ClickHouse/ClickStack#3477 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Slack notifications for inline alerts fail when the chart-explorer link passes Slack's 3000-character block limit対応中かも @Tyagiquamar が 28 日前に担当しました。 オープンalerting bug external
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
ClickHouse/ClickStack#3452 ·
メンテナーはふだん 1 日以内に返信
-
check-alerts worker: 13 error spans (mongodb.create, NamespaceExists) every minute because mongoose autoCreate is still on再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープンbug external Needs Feedback
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
ClickHouse/ClickStack#3308 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
count_distinct aggFn on metric tiles ignores valueExpression, always counts distinct metric Values対応中かも @Harshul1484 が 54 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
ClickHouse/ClickStack#2311 ·
メンテナーはふだん 1 日以内に返信
ClickHouse/ClickStack の issue をすべて見る
似ている issue
-
Add: GTrekkerオープンcheck:passed streams:add
難易度 1/5 1時間未満 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
area: ui Bug Payload OSS
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
payloadcms/payload#19839 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
メンテナーはふだん 1 日以内に返信
-
fix(auth): add the email OTP verification step to Pollar login対応中かも @akansnoro が今日担当しました。 オープンStellar Wave
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
safetrustcr/dApp-SafeTrust#584 · コメント 10 件 ·
メンテナーはふだん 1 日以内に返信
-
The manager preset saves launch definitions without the required file and digest pair対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
メンテナーはふだん 1 日以内に返信