chore: replace IVF split memory resource test with component tests

Abierto
#10,973 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
68/100
Tipo de issue
Refactorización
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
data, testing

Línea de trabajo

Start in rust/lance/src/index/vector/builder.rs at raw_vector_fetch_shape, take_vectors_stream, and slice_evenly; read their implementations and the existing regroup_by_bytes unit test. Add focused tests for fetch shape, bounded concurrent calls, and slice counts and sizes, then remove the slow resource test from PR CI. Done when the component tests cover the stated cases and the full scale check is assigned to a benchmark or nightly job.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

chore rust

test_ivf_split_reshuffle_memory_stays_bounded is the slowest test in the Rust suite: about 100s in the ci profile (timings from #10971). It guards the fix from #9574 by building an IVF-PQ index over 512 MiB of vectors, running optimize_indices to force a split, and checking peak allocations against a 128 MiB ceiling. Allocation tracking only works at that scale, since at small scale runtime overhead hides the signal.

The bound comes from a few small components in reshuffle_partitions, each with its own number we can assert directly:

https://github.com/lance-format/lance/blob/05a401f04406cf45ae9860d3415316bb79f45149/rust/lance/src/index/vector/builder.rs#L2422-L2435

Proposal: replace the resource test with sub-100ms tests of each component.

  • raw_vector_fetch_shape is pure and untested. Use rstest cases over block size (64 KiB, 4 KiB), dimension, and multivector vs fixed-size. Assert rows_per_fetch * row_bytes <= RAW_VECTOR_FETCH_BYTES and the in-flight count. This alone catches the original bug (block size in bytes used as a row count).
  • take_vectors_stream should take the fetch as a parameter instead of calling Dataset::take_rows itself. A test then passes a fake fetch that counts concurrent calls, and asserts it never exceeds prefetch. That catches a return to buffered(io_parallelism).
  • slice_evenly: assert the slice count and sizes. regroup_by_bytes already has a unit test.

Then move the full-scale test out of PR CI, either to a benchmark or a nightly job.

Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 22 h
PR fusionados (30 d)
572

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.