concurrent_qps measures swap-thrashing, not throughput, for single-process engines (introduced in #860)
維護者通常 1 天內回覆
評估
這個 Issue 還沒有評估資料。
描述
Summary
The concurrent_qps test introduced in #860 produces misleading, non-reproducible numbers for single-process / in-process engines (embedded CLIs and dataframe-server systems). For these systems it spawns N independent full-machine processes that collectively oversubscribe RAM, so on memory-constrained instances the metric measures swap-thrashing throughput, not query throughput — and whether it completes at all depends on the runner environment rather than the engine.
This surfaced in #943 (DataFusion), but the root cause is in lib/benchmark-common.sh (bench_concurrent_qps) and affects a whole class of systems.
Evidence
bench_concurrent_qps launches BENCH_CONCURRENT_CONNECTIONS (default 10) workers for BENCH_CONCURRENT_DURATION (default 600s). For BENCH_DURABLE=no engines like datafusion-cli, each worker's ./query spawns a fresh process that assumes it owns the whole machine. 10 × ~7 GB RSS ≈ 70 GB against 32 GB RAM.
Checked-in DataFusion (single) results, concurrent_qps vs. machine RAM:
| Machine | vCPU / RAM | concurrent_qps | err |
|---|---|---|---|
| c6a.xlarge | 4 / 8 GB | 0.025 | 0 |
| c6a.2xlarge | 8 / 16 GB | 0.042 | 0 |
| c6a.4xlarge | 16 / 32 GB | 0.11 | 0 |
| c8g.4xlarge | 16 / 32 GB | 0.092 | 0 |
| c6a.metal | 192 / 384 GB | 4.017 | 0 |
| c8g.metal-48xl | 192 / 384 GB | 5.388 | 0 |
| c7a.metal-48xl | 192 / 384 GB | 5.888 | 0 |
(datafusion-partitioned shows the same shape: 0.025–0.097 on the small boxes, 4.9–7.0 on the metals.)
The QPS tracks RAM, not core count — the metric on the sub-RAM machines is reporting how fast the box can page, not how fast the engine runs queries. err=0 everywhere, so nothing crashed in the automated runs; the processes just crawled through swap.
Why the same instance type gives different outcomes
cloud-init.sh.in adds a 16 GB swapfile and installs earlyoom on launch. In the automated fleet run a c6a.4xlarge therefore thrashes through swap and finishes at ~0.11 qps. A contributor running ./benchmark.sh on a bare c6a.4xlarge (no swap) gets the kernel OOM-killing datafusion-cli outright (see #943). So a published number depends on the presence of swap in the runner environment — a reproducibility problem independent of any one engine.
Scope
The same dynamic applies to every BENCH_DURABLE=no single-process / in-process system where N independent processes oversubscribe RAM — e.g. duckdb (and dataframe/memory variants), sqlite, hyper, chdb, pandas, polars-dataframe, daft, sirius. Any of these on a sub-RAM machine will show the same artifact.
Options
- Run the concurrent test at concurrency = 1 for single-process engines. Measures real serial throughput, stays comparable, never oversubscribes RAM. (Suggested by @alamb in #943.)
- Disable
concurrent_qpsfor the whole single-process / in-process class. #943 does this for DataFusion only (BENCH_CONCURRENT_DURATION=0); make it class-wide so the dashboard isn't uneven (one embedded engine reportingnullwhile peers report swap artifacts). - Add a "fit in RAM" guard analogous to the partial-load
>= 5 GBcheck: if the concurrent test's working set exceeds available RAM, recordnullinstead of a thrashing number. - Pin the environment: document/encode whether swap is part of the canonical setup so the same instance type reproduces the same result. (Weakest on its own — a number that only completes because of swap shouldn't be published regardless.)
Recommendation: option 1 or 2 for the single-process class (subsumes #943 into a consistent, class-wide fix), plus nulling the existing swap-artifact concurrent_qps values already checked in for these systems on the RAM-constrained machines.
Related
- #860 — introduced the concurrent QPS test
- #943 — DataFusion-specific workaround that prompted this
- 主要語言
- Shell
- 星號
- 1.2k
- 分支
- 334
- 平均合併
- 3 小時 50 分鐘
- 30 天內合併 PR
- 710
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 沒有貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
ClickHouse/ClickBench 的其他 Issue
-
難度 5/5 一週以上 新手友好度 12/100
ClickHouse/ClickBench#3294 ·
維護者通常 1 天內回覆
-
難度 4/5 3-5 天 新手友好度 25/100
ClickHouse/ClickBench#3292 ·
維護者通常 1 天內回覆
-
難度 4/5 3-5 天 新手友好度 15/100
ClickHouse/ClickBench#3291 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 5/5 一週以上 新手友好度 5/100
ClickHouse/ClickBench#3288 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 5/5 一週以上 新手友好度 35/100
ClickHouse/ClickBench#2650 ·
維護者通常 1 天內回覆
查看 ClickHouse/ClickBench 的全部 Issue
相似的 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 70/100
jeanibarz/knowledge-base-mcp-server#1268 ·
維護者通常 1 天內回覆
-
Unknown options to `omarchy capture screenrecording` open the capture picker可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉bug
難度 2/5 1-3 小時 新手友好度 72/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 62/100
OHDSI/Data2Evidence#3693 ·
維護者通常 3 天內回覆
-
難度 2/5 1-3 小時 新手友好度 62/100
cpp-best-practices/project_validator#260 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 72/100
jbaruch/coding-policy#818 ·
維護者通常 1 天內回覆