chore: replace IVF split memory resource test with component tests

Aperta
#10,973 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Refactoring
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
data, testing

Direzione di ricerca

Start in rust/lance/src/index/vector/builder.rs at raw_vector_fetch_shape, take_vectors_stream, and slice_evenly; read their implementations and the existing regroup_by_bytes unit test. Add focused tests for fetch shape, bounded concurrent calls, and slice counts and sizes, then remove the slow resource test from PR CI. Done when the component tests cover the stated cases and the full scale check is assigned to a benchmark or nightly job.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

chore rust

test_ivf_split_reshuffle_memory_stays_bounded is the slowest test in the Rust suite: about 100s in the ci profile (timings from #10971). It guards the fix from #9574 by building an IVF-PQ index over 512 MiB of vectors, running optimize_indices to force a split, and checking peak allocations against a 128 MiB ceiling. Allocation tracking only works at that scale, since at small scale runtime overhead hides the signal.

The bound comes from a few small components in reshuffle_partitions, each with its own number we can assert directly:

https://github.com/lance-format/lance/blob/05a401f04406cf45ae9860d3415316bb79f45149/rust/lance/src/index/vector/builder.rs#L2422-L2435

Proposal: replace the resource test with sub-100ms tests of each component.

  • raw_vector_fetch_shape is pure and untested. Use rstest cases over block size (64 KiB, 4 KiB), dimension, and multivector vs fixed-size. Assert rows_per_fetch * row_bytes <= RAW_VECTOR_FETCH_BYTES and the in-flight count. This alone catches the original bug (block size in bytes used as a row count).
  • take_vectors_stream should take the fetch as a parameter instead of calling Dataset::take_rows itself. A test then passes a fake fetch that counts concurrent calls, and asserts it never exceeds prefetch. That catches a return to buffered(io_parallelism).
  • slice_evenly: assert the slice count and sizes. regroup_by_bytes already has a unit test.

Then move the full-scale test out of PR CI, either to a benchmark or a nightly job.

Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.