Help reproducing the 100k/27s benchmark in #560: ingestion throughput and Redis CPU saturation

Start by reading issue #560 and its high-volume guide, then compare their benchmark script, payload and completion criteria with the workload details recorded here. Use the listed configuration and reconciliation criteria to frame a comparable reproduction, and gather measurements that separate worker processing, Redis buffer operations and ClickHouse insertion waits. Done means the baseline and resource bottleneck are characterized; the issue does not name specific code files or tests.

Summary

Thank you for #560 and the updated high-volume guide. We tested main commit 7f6f7f9e2b978bd0472807f586d65928f793d08c and would like help aligning our workload with the reported 100,000 events reaching ClickHouse in 27 seconds.

Our local tests take approximately 87–104 seconds for 100k events through HTTP, or 74–89 seconds replaying Kafka with the API stopped. A completed 1m-event Kafka replay takes 441.25 seconds (~2,266 business events/s). These are finite-run completion rates, not measured sustained capacity. We are reporting a performance investigation, not claiming a confirmed regression or an equivalent benchmark comparison.

Environment

• Intel i7-8700B host; Colima VM using macOS Virtualization.Framework, native x86_64, 6 vCPU / 24 GB shared.
• Each API and worker container: limit 2 CPU / 4 GB; these are caps, not dedicated cores.
• ClickHouse 25.10.2.65: limit 2 CPU / 8 GB.
• Redis 7.2.5-alpine: no CPU quota or CPU pinning; default io-threads=1, io-threads-do-reads=no.
• Redpanda v25.1.9: smp=1, configured memory 1 GB; 25 topic partitions.
• Some original infrastructure/dashboard containers remained running, while the original API/workers were paused. This was not a dedicated bare-metal capacity test.
• Application ingestion code was unchanged from the tested commit.

Test-only differences to disclose

The isolated test database's sessions sorting key includes id, to avoid cross-session collapsing under colliding creation timestamps. The event table was unchanged. This is a test schema variant, not a claim of unmodified upstream acceptance.

For successful Kafka replay controls we used Redpanda group_initial_rebalance_delay=5000 ms after observing duplicates during an initial rebalance with the original 0-ms setting. This is not an exactly-once guarantee or a general rebalance fix. Those correctness observations are being documented separately.

Workload and completion criteria

• Synthetic game-style events, approximately 20 custom properties, including device/account/role/server identifiers, sequence, level, channel, currency and an array property.
• 10,000 identities: 10 events each for 100k; 100 each for 1m.
• Counts/rates below refer to business events, excluding automatically generated session_start events.
• Each run used fresh ClickHouse/Redis volumes, verified empty before starting.
• HTTP tests: 2 API containers, 2 Node generator processes × 128 concurrent requests, single-event HTTP requests directly inside the Docker network; no Caddy/TLS. Timer starts after services are ready and ends after complete business events and all buffers drain.
• Replay tests: API/generators stopped; replay a fixed Kafka fixture into empty sinks with a fresh consumer group. Starting offsets are 0 across all 25 partitions and source totals are verified. Timer includes worker cold startup, group joining, and completion with Kafka committed lag and all buffers at zero.
• Historical completed test topics were cleaned before replay testing, reducing broker overhead. HTTP vs replay is therefore not a controlled comparison of API overhead alone.

Configuration

Selected configuration for the replay tests:

EVENT_BUFFER_BATCH_SIZE=10000
EVENT_BUFFER_CHUNK_SIZE=10000
SESSION_BUFFER_BATCH_SIZE=20000
# SESSION_BUFFER_CHUNK_SIZE is unset
PROFILE_BUFFER_BATCH_SIZE=500
PROFILE_BUFFER_CHUNK_SIZE=1000
CRON_CONCURRENCY=6
BUFFER_CH_INSERT_CONCURRENCY=5

This is a tested configuration, not a claim that it is globally optimal.

Results

HTTP, 100k events

Workers	Event batch / chunk	Completion time	Business events/s
2	10,000 / 2,000	90.4 s	1,106
2	10,000 / 10,000	87.3 s	1,146
2	10,000 / 10,000, repeat	92.9 s	1,076
2	50,000 / 10,000	96.8 s	1,033
3	10,000 / 10,000	104.4 s	958

Increasing batch size or replicas did not improve these shared-VM HTTP runs. The two 10k-chunk runs do not demonstrate a significant improvement over the 2k-chunk run.

Kafka replay, API/generators stopped

Events	Workers	Redis configuration	Completion time	Business events/s
100k	2	Default	89.10 s	1,122
100k	3	Default	74.21 s	1,348
100k	3	2 I/O threads, threaded reads enabled	77.83 s	1,285
100k	3	Default, repeat	79.46 s	1,258
1m	3	Default	441.25 s	2,266

All listed runs passed event/session/profile reconciliation. The 1m run had 1,000,000 business rows and unique sequences, 10,000 reconciled sessions with 100 events each, 10,000 profiles, and no remaining Kafka lag or buffer backlog. Redis I/O threading showed no clear gain within the observed variation.

Observations worth investigating

During sampled middle portions of the 1m replay:

• Redis averaged about 93% CPU, with 100% representing one CPU core.
• Each worker averaged approximately 103–108% CPU.
• ClickHouse averaged about 46% CPU, with a sampled peak of about 160% against its 200% cap. This does not exclude transient ClickHouse or I/O waits.
• At 390.64 s, Kafka lag was first observed at zero; 640k business events had landed, with about 360k event-buffer rows and 350k session-buffer rows pending.
• The remaining 360k business events landed and all buffers drained in about 50.61 s (~7,114 business events/s during this tail only).

The tail suggests draining can be faster once upstream processing pressure subsides; it does not establish sustained ingestion capacity. Redis CPU is a lead, not proof that Redis alone is the bottleneck. We do not yet have a CPU profile proving the limiting code path.

Questions

1. Could you share the exact script/command and payload used for the 100k/27s result in #560, including payload size, identity/session cardinality, and event mix?
2. What hardware, runtime version, container limits, Kafka/Redis settings, and ClickHouse schema/materialized views were used?
3. Does the 27-second completion criterion mean business events visible in ClickHouse only, or also drained session/profile buffers? Does timing include startup or group joining?
4. For this workload, which measurements would you recommend next to distinguish worker processing, Redis buffer operations and ClickHouse insertion waits?
5. With the current per-buffer locking/drain design, what worker/Redis/ClickHouse deployment and tuning would you recommend for sustained 5k+ business events/s while keeping session/profile backlog bounded?

We would like to reproduce the baseline first, then provide comparable measurements rather than attributing the difference to a particular component prematurely.