Reader: concurrent scan with a confined arena per worker thread

Abierto
  • ¿Algo no está bien?
#505 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
12/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java
Área
data, performance

Línea de trabajo

This is a design proposal, so the first step is deciding the open questions, not editing code. Read ScanIterator, Chunk and the existing confined-versus-shared arena handling, plus ADR 0003 for the exception contract on worker threads. Done means a recorded decision on shared versus per-worker mapping and ordered versus unordered delivery, then an A/B run with RealFileScanBenchmark.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement perf

Parked design for a concurrent reader scan. Follow-up to #474 (writer side, shipped).

Reference behavior

  • Rust (vortex-layout/src/scan/scan_builder.rs): the scan is split into row-range tasks (SplitBy), each spawned onto the session runtime, up to concurrency * num_workers in flight (default 4 per worker). ordered (default) uses .buffered(n), otherwise .buffer_unordered(n). A per-split map function runs on the spawned task.
  • .NET (Vorticity): MaxDegreeOfParallelism per session or per scan, MaxConcurrentReads for I/O, shared SegmentCache, concurrent scans over one open file.
  • vortex-java today: ScanIterator is single-threaded, every arena is ofConfined.

Proposed shape

A confined Arena per worker thread, so the callback runs on the worker and the caller never touches decoded memory from another thread:

scan.parallel(executor)            // executor + bounded in-flight window on ScanOptions
    .map(chunk -> r)               // runs on the worker; chunk and its arena close on return
  • R must not reference chunk memory (aggregates, copied-out values); Javadoc states the contract.
  • Results delivered ordered (Rust buffered, default) or unordered (buffer_unordered).
  • The file mapping becomes ofShared (one close at the end), since workers must read it. Alternative: each worker maps its own view.
  • scan() / ScanIterator / Chunk lifetime rules stay unchanged.

Pros

  • Rust-shaped (map on the spawned task, ordered/unordered, bounded window), so parity with the reference.
  • Confined arenas keep the cheap alloc/close path; Arena.ofShared().close() needs a cross-thread handshake per chunk.
  • No change to the existing single-threaded API or its safety checks.
  • Same bounded-window idea as the writer's pending queue / MAX_IN_FLIGHT_SEGMENTS.
  • Overlaps decode with consumption; wide files and slow consumers benefit most.

Cons

  • No ordered, parallel, zero-copy Iterator<Chunk> for a consumer thread: a chunk decoded on a worker cannot be read elsewhere. That case would need a shared per-chunk arena (slower close, and relaxes the "second next() while a Chunk is open is rejected" rule).
  • Callback API is less ergonomic than an iterator; callers who must keep data have to copy out.
  • Mapping arena goes shared; any cross-thread use after close() becomes a hazard to test (ADR 0003 security contract still applies on worker threads: only VortexException may escape).
  • Worker exceptions must be propagated and in-flight work cancelled/drained on failure (the writer's discardPending equivalent).
  • Needs A/B on all 7 real files (RealFileScanBenchmark, interleaved): memory-bandwidth-bound scans may not scale, and oversubscribing the pool gains nothing.

Out of scope / not planned

  • Per-column parallel decode within a chunk: not a Rust pattern, so it would be a deliberate divergence needing an explicit decision.
  • Parallel I/O for VortexHttpReader (concurrent range requests): independent of this, worth its own ticket if pursued.
  • Writer follow-ups (parallel scheme trials in CascadingCompressor.chooseBest): comment on #474.
Lenguaje dominante
Java
Estrellas
3
Forks
2
Merge medio
9 h 47 min
PR fusionados (30 d)
128

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dfa1/vortex-java

Todos los issues de dfa1/vortex-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.