flaky: ruvector-quantization 'should perform search with binary quantization' recall floor fails on unseeded random vectors

Open Beginner friendly
#3,207 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
testing

Research direction

Start with v3/@claude-flow/plugins/tests/ruvector-quantization.test.ts, especially the binary-quantization test at line 493 and the related perturbation case near line 508. Run npx vitest run __tests__/ruvector-quantization.test.ts, then make the recall check deterministic or base it on the planted near-duplicate scenario described in the issue. Done means the test passes reliably without masking regressions or depending on unseeded random vectors.

Written by the indexing model from the issue text.

Description

Observed: ruflo CI run 33974289751 (PR #3184 at 8b3927d, ubuntu Test Suite) — the CI test ratchet flagged one unexpected failing file, v3/@claude-flow/plugins/__tests__/ruvector-quantization.test.ts:

RuVector Quantization > Binary Quantization > should perform search with binary quantization
AssertionError: expected 0 to be greater than or equal to 0.1
  at __tests__/ruvector-quantization.test.ts:493:22

The PR does not touch that file (diff vs main: LEDGER, gist, cli-cold-start test/bench, ADR status summary). The file is byte-identical to main, and locally it passes 28/28 (npx vitest run __tests__/ruvector-quantization.test.ts). It also passed on #3204's run 33972961703 minutes earlier on the same base.

Why it flakes: the test builds its corpus with randomVector(dimensions) (unseeded Math.random) and asserts binary-quantized top-k recall ≥ 0.1 against exact search. Binary quantization over i.i.d. random vectors has no such floor for small k — recall of 0 is a legitimate outcome of the RNG, not a regression. Sibling tests in the file have the same shape (line 508 perturbs with Math.random() too).

Ask (either):

  1. seed the generator (a small LCG/xorshift randomVector(dim, seed)) so the corpus is fixed and the assertion is deterministic, or
  2. assert a property binary quantization actually guarantees (e.g. recall over a corpus with a planted near-duplicate, as line 508 does), and drop the 0.1 floor on random data.

Until then the ratchet will keep failing unrelated PRs at random; I re-ran the failed job on #3184 rather than touching the baseline, since adding it to scripts/ci-test-baseline.txt would hide a real regression in that file.

🤖 Generated with RuFlo

https://claude.ai/code/session_019xHM4rAH4aaShb4DTr1n6s

Dominant language
TypeScript
Stars
72.9k
Forks
8.7k
Avg merge
3d 10h
Merged PRs (30d)
74

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruvnet/ruflo

All issues in ruvnet/ruflo

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.