Reproducible span-ingestion benchmark and live capacity qualification

Open
  • Is something wrong?
#3,674 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
1/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Stale
Tech stack
clickhouse, python
Domain
backend, observability, testing-qa

Research direction

Start with PR #3572, which contains the maintainer-requested implementation, and inspect its benchmark and integration test entry points. Compare the work with the acceptance checklist, beginning with real collector-to-ClickHouse ingestion and independent trace/span ID accounting. Done requires the listed live, endurance, burst, overload, and review evidence; the issue says these targets are not yet asserted results.

Written by the indexing model from the issue text.

Description

accepted

Problem

We need a reproducible way to measure the real OTLP receiver → collector/converter → ClickHouse path, rather than treating HTTP acknowledgements or a direct-storage seeder as ingestion capacity evidence.

This records the scope and outstanding acceptance for the existing maintainer-requested implementation in PR #3572 (tracking key TH-8687). Accepting the scope is not code approval, a passing benchmark result, or authorization to merge.

Implementation scope

  • Deterministic synthetic workloads with configurable payload sizes, spans per trace and traces per session.
  • Separate offered, scheduled, attempted, acknowledged, stored and read-visible counts/timings.
  • Independent trace/span-ID reconciliation for missing, duplicate and unexpected data.
  • Bounded execution, explicit target/load confirmation, telemetry, checkpoints and reproducible report inputs.
  • Truthful integration with the collector's current byte-based admission and resource-ownership model; row counts are not memory guarantees.
  • Strict validation of receipt counters, sampling evidence and byte-pool partitions. Inconsistent evidence must fail report/publication rather than produce a plausible report.
  • Genuine opt-in ClickHouse integration tests and CI smoke; offline fixtures must remain labelled as fixtures.

Verification and acceptance

  • Integrate current dev while preserving authentication, resource leases, writer ownership and shutdown/drain safeguards.
  • Pass unit, race, vet/build, environment-registry, contract/coverage and applicable hosted CI checks at the final head.
  • Exercise the real collector/ClickHouse path and verify independent ID accounting; skipped live tests are not a pass.
  • Run an authenticated deployed smoke and bounded pilot against isolated synthetic test infrastructure.
  • Capture a genuine sustained 15,000 spans/s run for 24 hours and a separate 60,000 spans/s burst, with offered-rate and producer-delay evidence.
  • Verify missing/duplicate/unexpected IDs and backlog/visibility behavior, and record application read timings and resource/storage amplification.
  • Exercise controlled overload/recovery and explicitly distinguish in-memory acknowledgement from durable-before-ack guarantees.
  • Qualify any replicated failure/recovery claims on the actual tested topology; do not infer them from a single-node smoke.
  • Obtain independent non-author review before merging.

Evidence boundaries

Targets above are not asserted results. A small smoke, offline receipt, sizing formula or extrapolated pilot cannot stand in for endurance or production capacity. If the harness lands before full capacity qualification, retain the unfinished acceptance work explicitly rather than marking TH-8687 complete. The existing collector does not gain a durable pre-ack queue merely because this benchmark measures it.

Implementation and current evidence: https://github.com/future-agi/future-agi/pull/3572

Dominant language
Python
Stars
2k
Forks
628
Avg merge
2d 2h
Merged PRs (30d)
259

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from future-agi/future-agi

All issues in future-agi/future-agi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.