HTTPS pool: profile the task-queue lock and compare against sharded / lock-free variants
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- c, linux
- Domain
- backend, performance
Research direction
Start by reading scripts/ci/c100k_https.sh and the HTTPS pool implementations of https_pool_submit_ready(), https_pool_submit_conn() and the worker pop. Profile the ramp, request rounds and RSS-gate load with perf record -g, then measure lock hold times at 1, 4 and 12 workers. Compare variants on the listed scenarios; done means reporting results and considering a variant only if it shows a measurable scenario-level win and passes TSan, RSS-gate and C100k.
Written by the indexing model from the issue text.
Description
Follow-up to the r/cprogramming thread about replacing the HTTPS pool's ring buffer with a lock-free queue. The thread's consensus is to measure before changing the design, and the server is mostly I/O-bound, so this issue records the measurements to make. The resize-under-mutex point raised there is fixed in 83fbff48: the new ring is allocated, and the old one freed, outside g_https_pool.mutex.
Profile first
perf record -gonscripts/ci/c100k_https.sh(ramp and request rounds) and on the RSS-gate load. Report how much time is spent inpthread_mutex_lock/futexong_https_pool.mutex, against time in TLS, syscalls and the handler.- Measure lock hold times for
https_pool_submit_ready(),https_pool_submit_conn()and the worker pop, usingperf lockor bpftrace onfutex. Do this at 1, 4 and 12 workers.
Scenarios to compare (same machine, same build flags)
| Scenario | What it stresses |
|---|---|
| C100k ramp (connect burst, 512 in-flight handshakes per client thread) | submit-side contention, ring growth |
| C100k keep-alive rounds (100k requests in flight) | park/unpark resubmits through https_pool_submit_conn() |
Connection churn (ab -c 1000, no keep-alive) |
handshake shards to pool handoff |
Small pool (CWIST_WORKERS=1, one core, like oborona.zip) |
queue depth and waiting, not contention |
Variants
- Current mutex + condvar ring, with growth outside the lock (baseline).
- Per-shard rings (one mutex per N workers), only if (1) shows real contention.
- A lock-free MPMC ring, only if (2) still shows contention.
Each variant would need TSan, RSS-gate and C100k results before it is considered. The bar for (2) or (3) is a measurable win on at least one scenario above, not on a microbenchmark.
- Dominant language
- C
- Stars
- 16
- Forks
- 7
- Avg merge
- 4h 28m
- Merged PRs (30d)
- 191
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from c4punks/CWIST
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
-
enhancement good first issue
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
-
enhancement good first issue
Difficulty 4/5 3-5 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 57/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
Similar issues
-
kind/bug priority/medium qa_automation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
yugabyte/yugabyte-db#37324 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
meshcore-dev/MeshCore#3781 ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
ly05010419/MacNativeGames#1897 ·
Maintainers usually reply within 1 day
-
area: metrics area: tracing contribfest enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
open-telemetry/opentelemetry-ebpf-instrumentation#4188 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day