concurrent_qps measures swap-thrashing, not throughput, for single-process engines (introduced in #860)

Offen
  • Stimmt etwas nicht?
#946 6 Kommentare 1 Reaktion 1 zugewiesene Person Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@george-larionov arbeitet bereits daran.

Seit 15.6.2026.

  • #3109 von @alexey-milovidov — offen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Beschreibung

Summary

The concurrent_qps test introduced in #860 produces misleading, non-reproducible numbers for single-process / in-process engines (embedded CLIs and dataframe-server systems). For these systems it spawns N independent full-machine processes that collectively oversubscribe RAM, so on memory-constrained instances the metric measures swap-thrashing throughput, not query throughput — and whether it completes at all depends on the runner environment rather than the engine.

This surfaced in #943 (DataFusion), but the root cause is in lib/benchmark-common.sh (bench_concurrent_qps) and affects a whole class of systems.

Evidence

bench_concurrent_qps launches BENCH_CONCURRENT_CONNECTIONS (default 10) workers for BENCH_CONCURRENT_DURATION (default 600s). For BENCH_DURABLE=no engines like datafusion-cli, each worker's ./query spawns a fresh process that assumes it owns the whole machine. 10 × ~7 GB RSS ≈ 70 GB against 32 GB RAM.

Checked-in DataFusion (single) results, concurrent_qps vs. machine RAM:

Machine vCPU / RAM concurrent_qps err
c6a.xlarge 4 / 8 GB 0.025 0
c6a.2xlarge 8 / 16 GB 0.042 0
c6a.4xlarge 16 / 32 GB 0.11 0
c8g.4xlarge 16 / 32 GB 0.092 0
c6a.metal 192 / 384 GB 4.017 0
c8g.metal-48xl 192 / 384 GB 5.388 0
c7a.metal-48xl 192 / 384 GB 5.888 0

(datafusion-partitioned shows the same shape: 0.025–0.097 on the small boxes, 4.9–7.0 on the metals.)

The QPS tracks RAM, not core count — the metric on the sub-RAM machines is reporting how fast the box can page, not how fast the engine runs queries. err=0 everywhere, so nothing crashed in the automated runs; the processes just crawled through swap.

Why the same instance type gives different outcomes

cloud-init.sh.in adds a 16 GB swapfile and installs earlyoom on launch. In the automated fleet run a c6a.4xlarge therefore thrashes through swap and finishes at ~0.11 qps. A contributor running ./benchmark.sh on a bare c6a.4xlarge (no swap) gets the kernel OOM-killing datafusion-cli outright (see #943). So a published number depends on the presence of swap in the runner environment — a reproducibility problem independent of any one engine.

Scope

The same dynamic applies to every BENCH_DURABLE=no single-process / in-process system where N independent processes oversubscribe RAM — e.g. duckdb (and dataframe/memory variants), sqlite, hyper, chdb, pandas, polars-dataframe, daft, sirius. Any of these on a sub-RAM machine will show the same artifact.

Options

  1. Run the concurrent test at concurrency = 1 for single-process engines. Measures real serial throughput, stays comparable, never oversubscribes RAM. (Suggested by @alamb in #943.)
  2. Disable concurrent_qps for the whole single-process / in-process class. #943 does this for DataFusion only (BENCH_CONCURRENT_DURATION=0); make it class-wide so the dashboard isn't uneven (one embedded engine reporting null while peers report swap artifacts).
  3. Add a "fit in RAM" guard analogous to the partial-load >= 5 GB check: if the concurrent test's working set exceeds available RAM, record null instead of a thrashing number.
  4. Pin the environment: document/encode whether swap is part of the canonical setup so the same instance type reproduces the same result. (Weakest on its own — a number that only completes because of swap shouldn't be published regardless.)

Recommendation: option 1 or 2 for the single-process class (subsumes #943 into a consistent, class-wide fix), plus nulling the existing swap-artifact concurrent_qps values already checked in for these systems on the RAM-constrained machines.

Related

  • #860 — introduced the concurrent QPS test
  • #943 — DataFusion-specific workaround that prompted this
Vorherrschende Sprache
Shell
Sterne
1.2k
Forks
334
Ø Merge
3 Std. 50 Min.
Gemergte PRs (30 T.)
710

Entwicklungsumgebung

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus ClickHouse/ClickBench

Alle Issues in ClickHouse/ClickBench

Ähnliche Issues

Weitere Issues zu Shell/Bash

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.