No gate asserts Tree.of build cost at the shipped ruleset size

Open
  • Is something wrong?
#768 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java
Domain
build-system, ci-cd, testing-qa

Research direction

Start with docs/docs/deploy/operations.md, TreeBuildBenchmark, TreeBuildWorkCounterTest, and .github/scripts/check-build-cost-docs.sh. Run git log --grep='^Cost-Unchanged:' --oneline | wc -l and inspect make bench-jmh BENCH_TARGET=TreeBuildBenchmark against the published 262,251,844 work count. Done means the remaining shipped-ruleset build-cost gap has an agreed, reproducible gate or a documented decision not to add one.

Written by the indexing model from the issue text.

Description

enhancement

Rewritten 2026-09-07. Most of the original text is no longer true: #770 and #772 have landed, and the two closures this issue proposed both turned out to be dead. The original is preserved at the bottom, because the .parallel() demonstration in it is what justified everything that followed.

Split out of #746 rather than folded into it, because closing #746 was a documentation decision and this is a missing gate.

What is still open

Build cost is unasserted at the shipped ruleset size. docs/docs/deploy/operations.md publishes a work count of 262,251,844 for the bundled ruleset and a wall time at four sizes. No test asserts any of it. The work counter added by #770 pins 284,792 and 1,124,740, which are the first 200 and 400 CSV rows — a 31x extrapolation away from the 12,496 preprocessed rules the docs publish.

That is the whole of what remains. Everything else this issue was filed about has been closed.

What closed, and how

The figures no longer go stale silently. Three observers now name operations.md, the section, the figure and the re-measure command when they red:

  • TreeBuildWorkCounterTest — on any change to the verdict count (#772)
  • BundledRulesetTreeIdentityTest — on any ruleset edit, including a value-only one that holds row and distinct-port counts constant, which is the only test that sees that case (#772)
  • TreeBuildSynthesisTest — on a change to the row counts, which it already did

The .parallel() demonstration is now caught. .github/scripts/check-build-cost-docs.sh refuses a diff touching classification/internal/decision/ unless operations.md changed with it or a commit carries a Cost-Unchanged: trailer with a reason. Verified against this issue's own demonstration: deleting .parallel() is refused, where every existing test stays green.

A cost regression at small scale now fails an assertion. WORK_AT_SLICE and WORK_AT_DOUBLE_SLICE are exact, not ratio-with-headroom, because the quantity is a count rather than a duration.

Both closures this issue proposed are dead

"A budget row in src/bench." bench is invoked by no workflow. Every make target any workflow runs: e2e, jar, packages, packages-smoke, compose-smoke, docs, docs-lint-test, fuzz, nix, nix-hash, nix-check, lint-actions, contributors-check, release, release-lineage, release-lineage-test, sbom-assert, sbom-assert-test, and now build-cost-docs/build-cost-docs-test. The three ratio budgets in src/bench gate a target nothing runs.

"A cheap proxy: Tree.Info already carries sumComp and node counts." Already implemented — all twelve fields are pinned — and this issue's own .parallel() demonstration left every one of them green. The open question the original recorded ("whether Tree.Info fields actually track cost closely enough... plausible from the mechanism; unmeasured") is now measured and answered no: across the published doublings, work moves 4.00x while avgComp moves 1.08x, and for a process-only change shape does not move at all. Tree.Info is a function of the output; build cost is a property of the process.

The trigger

Build the gate when either is true:

  1. Cost-Unchanged: has been claimed on three or more merged commits. Check with git log --grep='^Cost-Unchanged:' --oneline | wc -l. Reads 0 as of 2026-09-07.
  2. A re-measurement finds the published bundled work count moved by more than 20% with no intended algorithmic change. Check against 262,251,844 after make bench-jmh BENCH_TARGET=TreeBuildBenchmark.

Condition 1 does not measure drift. A trailer is used for renames and comment edits too. It measures how often the question arises — if the build changes often enough that people routinely assert the cost did not move, a cheap prompt is carrying weight a real gate should carry. If it stays at 0 for a year, a gate would have been overhead, and that is the outcome to close this on.

Three is a judgment, not a measurement. #771's research found no published account, in any ecosystem, of what a deterministic cost gate costs to maintain over a year, so there is nothing better to calibrate against.

What is still NOT established

Whether a time-based budget is stable enough for CI. Unchanged, and now recorded as rejected rather than open: the build is core-count dependent and runners vary.

Whether the paired-file check is honoured or routed around. It is satisfiable by editing the document, and editing a document is not measuring. It converts a silent staleness into a prompt, and a prompt can be answered badly. The trailer only makes a bad answer auditable afterwards.

Whether the 200-row slice extrapolates. Anything conditioned on ruleset size hides in the 31x gap between the pin and the published size.

References

  • #770 — the work counter
  • #771 — the staleness-trigger proposal, with the evidence for everything above
  • #772 — the two staleness messages and the CI check
Original body, as filed 2026-09-06 (superseded above; kept for the .parallel() demonstration)

Split out of #746 rather than folded into it, because closing #746 was a documentation decision and this is a missing gate.

What is verified

docs/docs/deploy/operations.md now publishes a supported ruleset size and a build time per size — 929 ms at the shipped 6,248 rules, 4.43 s at the 12,496-rule bound. Nothing checks any of it.

  • BundledRulesetTreeIdentityTest pins twelve Tree.Info fields and a SHA-256 over 123,872 classification answers. It observes shape, never cost. Both its rows carry @Timeout(5, MINUTES), roughly 5× the largest published figure.
  • TreeBuildSynthesisTest counts rules, ports and preprocessed rules. Its own javadoc says "None of this builds a tree".
  • TreeBuildBenchmark measures the real thing, but Makefile states the JMH run "is opt-in and never a build gate", and grep -rn "bench" .github/workflows/*.yml returns nothing.

Demonstrated, not argued. Reviewing #767, deleting .parallel() at Tree.java:68 — a same-shape, different-cost change the file's own comment predicts is answer-neutral — left BundledRulesetTreeIdentityTest fully green, both the shape row and the answer digest. A change moving this build several-fold in either direction fails no assertion.

Why it matters

The figures are an operator-facing support commitment. Tree.of can be made materially slower and the docs keep asserting 929 ms and a 12,500-rule bound, with nothing telling a reader the numbers no longer describe the code. #767 added a provenance stamp (date, commit, machine) so a reader can at least see which Tree.of was measured — that is a mitigation, not a gate.

This is the same shape as the gap #765 closed for rejected rules: a real property with no observable behind it.

What would close this

The repo already has both mechanisms; this needs one of them pointed at Tree.of:

  • A budget row in src/bench (make bench, Makefile:119-136) — that harness exists precisely for ratio assertions, and a loose budget on one bundled build would catch a several-fold regression without making a multi-second build part of mvn verify.
  • Or a cheap proxy in the suite: Tree.Info already carries sumComp and node counts, which move with the work done. Pinning a proxy is weaker than timing but costs nothing and would catch an algorithmic change.

What is NOT established

  • Whether a time-based budget is stable enough for CI. The published numbers are core-count dependent — Tree.of scores candidates on a parallel stream — and CI runners vary. That is the reason a ratio or a proxy is likely better than an absolute millisecond budget, and it is the thing to settle before implementing.
  • Whether Tree.Info fields actually track cost closely enough to be a useful proxy. Plausible from the mechanism; unmeasured.
Dominant language
Java
Stars
2
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Riptide-Labs/riptide

All issues in Riptide-Labs/riptide

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.