Reader: concurrent scan with a confined arena per worker thread

Đang mở
  • Có gì không ổn?
#505 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
12/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
data, performance

Hướng nghiên cứu

This is a design proposal, so the first step is deciding the open questions, not editing code. Read ScanIterator, Chunk and the existing confined-versus-shared arena handling, plus ADR 0003 for the exception contract on worker threads. Done means a recorded decision on shared versus per-worker mapping and ordered versus unordered delivery, then an A/B run with RealFileScanBenchmark.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement perf

Parked design for a concurrent reader scan. Follow-up to #474 (writer side, shipped).

Reference behavior

  • Rust (vortex-layout/src/scan/scan_builder.rs): the scan is split into row-range tasks (SplitBy), each spawned onto the session runtime, up to concurrency * num_workers in flight (default 4 per worker). ordered (default) uses .buffered(n), otherwise .buffer_unordered(n). A per-split map function runs on the spawned task.
  • .NET (Vorticity): MaxDegreeOfParallelism per session or per scan, MaxConcurrentReads for I/O, shared SegmentCache, concurrent scans over one open file.
  • vortex-java today: ScanIterator is single-threaded, every arena is ofConfined.

Proposed shape

A confined Arena per worker thread, so the callback runs on the worker and the caller never touches decoded memory from another thread:

scan.parallel(executor)            // executor + bounded in-flight window on ScanOptions
    .map(chunk -> r)               // runs on the worker; chunk and its arena close on return
  • R must not reference chunk memory (aggregates, copied-out values); Javadoc states the contract.
  • Results delivered ordered (Rust buffered, default) or unordered (buffer_unordered).
  • The file mapping becomes ofShared (one close at the end), since workers must read it. Alternative: each worker maps its own view.
  • scan() / ScanIterator / Chunk lifetime rules stay unchanged.

Pros

  • Rust-shaped (map on the spawned task, ordered/unordered, bounded window), so parity with the reference.
  • Confined arenas keep the cheap alloc/close path; Arena.ofShared().close() needs a cross-thread handshake per chunk.
  • No change to the existing single-threaded API or its safety checks.
  • Same bounded-window idea as the writer's pending queue / MAX_IN_FLIGHT_SEGMENTS.
  • Overlaps decode with consumption; wide files and slow consumers benefit most.

Cons

  • No ordered, parallel, zero-copy Iterator<Chunk> for a consumer thread: a chunk decoded on a worker cannot be read elsewhere. That case would need a shared per-chunk arena (slower close, and relaxes the "second next() while a Chunk is open is rejected" rule).
  • Callback API is less ergonomic than an iterator; callers who must keep data have to copy out.
  • Mapping arena goes shared; any cross-thread use after close() becomes a hazard to test (ADR 0003 security contract still applies on worker threads: only VortexException may escape).
  • Worker exceptions must be propagated and in-flight work cancelled/drained on failure (the writer's discardPending equivalent).
  • Needs A/B on all 7 real files (RealFileScanBenchmark, interleaved): memory-bandwidth-bound scans may not scale, and oversubscribing the pool gains nothing.

Out of scope / not planned

  • Per-column parallel decode within a chunk: not a Rust pattern, so it would be a deliberate divergence needing an explicit decision.
  • Parallel I/O for VortexHttpReader (concurrent range requests): independent of this, worth its own ticket if pursued.
  • Writer follow-ups (parallel scheme trials in CascadingCompressor.chooseBest): comment on #474.
Ngôn ngữ chính
Java
Star
3
Fork
2
Merge trung bình
9 giờ 47 phút
Pull request đã merge (30 ngày)
128

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của dfa1/vortex-java

Tất cả issue của dfa1/vortex-java

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.