V2 ingestion: limited multi-worker throughput scaling and session buffer backlog

Start by reading the buffer implementations in packages/core/src/buffers, especially event and session flush behavior, locking, and batching; then compare their scheduling and metrics. The issue presents performance observations and possible sharding as discussion options, but does not settle on a change or define acceptance criteria. Done would require maintainers to agree on a scaling direction and its ordering, consistency, and migration requirements.

Summary

Thanks for the recent ingestion and buffering improvements.

We tested the updated main branch with one million events per run, using the Kafka ingestion path. We observed limited throughput scaling when increasing the number of workers. In some configurations, the event buffer drained while a substantial session buffer backlog remained.

Increasing the session batch size and adjusting cron concurrency improved backlog clearance. We would like to understand the intended scaling strategy and whether further buffer-level parallelism is planned.

This is a performance investigation and design discussion, not a claim about OpenPanel's maximum throughput.

Version

Benchmark source revision:

a887576a (main snapshot taken on 2026-10-18)

We also reviewed the buffer implementation at:

1e5bfa6a53e0d595a9f1c1026e0d03f7ee6afbeb

The buffer implementation was unchanged between these two snapshots.

Environment

All services ran in a shared local VM:

• VM: 6 vCPU, 24 GiB RAM
• API: 2 instances, each limited to 2 CPU / 4 GiB RAM
• Workers: 1–3 instances, each limited to 2 CPU / 4 GiB RAM
• ClickHouse: limited to 2 CPU / 8 GiB RAM
• Kafka-compatible broker: Redpanda
• Topic partitions: 25
• Shared Redis for buffering and application state
• ClickHouse client compression disabled for these comparison runs

Pipeline:

API → Kafka/Redpanda → Workers → Redis buffers → ClickHouse

The container CPU limits were not dedicated CPU reservations. Because all services shared the VM, resource contention may have affected the scaling results.

Test workload and measurement

Each run processed 1,000,000 synthetic events, including session processing.

The rates below describe the measured event-processing/drain phase. They should not be interpreted as HTTP request throughput or as proof of sustained production capacity.

We also inspected Kafka lag, Redis buffer backlogs, and ClickHouse event counts.

This was a finite-load test, not a multi-hour steady-state benchmark. The results may depend on device/session distribution and the workload's derived-state update rate.

Results

Configuration Approx. event throughput Approx. event completion time

1 worker 2,159 events/s 463 s

2 workers 3,038 events/s 329 s

3 workers 2,853 events/s 350 s

2 workers, larger event batches 3,010 events/s 332 s

2 workers, larger event and session batches, cron concurrency adjusted 2,969 events/s 337 s

For the larger-event-batch configuration:

  EVENT_BUFFER_BATCH_SIZE=50000
  EVENT_BUFFER_CHUNK_SIZE=10000
  BUFFER_CH_INSERT_CONCURRENCY=5

For the final configuration, we additionally used:

  SESSION_BUFFER_BATCH_SIZE=50000
  SESSION_BUFFER_CHUNK_SIZE=10000
  CRON_CONCURRENCY=2

Backlog observations

With the larger event batches alone:

• The session buffer still contained approximately 726,150 entries when event processing completed.
• Peak observed session buffer length was approximately 1,253,700.
• Observed Redis memory peaked at approximately 967 MiB.

With the additional session and cron tuning:

• Kafka lag and all checked buffer backlogs were zero at the completion checkpoint.
• ClickHouse event count and unique event count both matched the 1,000,000-event input.
• No errors were observed in that run.
• Peak observed session buffer length was approximately 87,550.
• Observed Redis memory peaked at approximately 204 MiB.

Session buffer entries represent state/version records, not distinct sessions or business events.

The main improvement was more balanced end-to-end drainage and lower backlog/memory usage, rather than higher event throughput.

Relevant implementation observations

From reviewing packages/core/src/buffers:

1. Shared queue and lock per buffer type

   Event writes converge on event_buffer:queue, coordinated by the event buffer lock.

   Multiple workers can process upstream work, but they do not create independent event-buffer flush lanes.

   Different buffer types have separate locks; this is not one global lock for all buffers.

2. Concurrency within a flush already exists

   Event buffer chunks use bounded concurrent ClickHouse inserts.

   Therefore, the current implementation is not strictly single-request-at-a-time. The question is whether the shared flush boundary limits further scaling.

3. Drain behavior differs by buffer type

   EventBuffer.processBuffer() processes one batch per invocation.

   Several other buffers use a time-budgeted loop to process multiple batches.

4. Session writes have different processing characteristics

   Session updates produce version/sign records, and the buffer squashes intermediate versions before insertion.

   Session chunks are currently inserted sequentially, rather than through the base buffer's bounded-concurrency helper.

5. Session phase-level visibility is limited

   Session flushing reports processed rows, but does not currently expose the same detailed read/insert/trim timings as some other buffers.

These observations are possible contributors, not a confirmed root-cause diagnosis. Shared VM contention, Redis, ClickHouse, scheduling, and workload characteristics may also explain part of the results.

Questions

1. What is the recommended configuration and deployment strategy for scaling sustained ingestion throughput across multiple workers?
2. Is the shared per-type buffer/lock an intentional coordination boundary? Are there ordering or consistency requirements that would make sharding unsafe?
3. Are there plans to improve event/session flush scheduling, add optional buffer sharding, or provide a documented multi-worker benchmark?
4. Would additional session flush phase metrics be useful for distinguishing parsing/squashing overhead from ClickHouse insert latency?

Possible direction for discussion

Would an opt-in, fixed-shard buffer design be compatible with the current architecture?

For example:

• Keep the current behavior as the default.
• Introduce independent Redis lists and locks per shard.
• Route records deterministically, preserving the required entity/session ordering.
• Allow workers to process different shards concurrently.
• Bound total ClickHouse insert concurrency per worker, rather than multiplying concurrency independently across every shard.
• Provide an explicit transition/drain procedure for existing queues.

This is only a proposal for discussion. We are not assuming sharding is the only solution, or that it would remove Redis/ClickHouse resource limits.

Our goal is to understand how to scale the existing ingestion architecture while preserving its current event and session semantics.