feat: automatic JSON shredding and transparent document access

Đang mở
#832 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python, rust

Hướng nghiên cứu

Read the remaining roadmap item in this issue: background re-shredding, workload benchmarks, and query-aware index and selection policy; array-path extraction is explicitly evidence-driven. Start by reviewing the merged PRs #833, #835, #836, and #839 to understand the current DocumentStore and Lance primitives. Define a bounded follow-up increment with tests and benchmarks; it is done when that increment's behavior is validated and its English-language PR passes CI.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Problem

Large JSON documents often contain one or two text fields that dominate their size. Keeping a complete JSON document plus extracted text columns duplicates those values. Users should not have to declare extraction paths, manage residual documents, or reconstruct records themselves.

Goal

Implement real automatic JSON shredding in lance-context: discover useful fields from incoming documents, choose typed physical columns automatically, remove extracted values from the residual, and expose the original logical document through reads and queries. Keep policy, reconstruction, and layout evolution in lance-context; use existing Lance storage/indexing primitives and avoid upstream changes unless a demonstrated gap requires a small fix.

Design requirements

  • No user-maintained list of shredded paths. Bound profiling, column growth, and memory use; account for value bytes as well as frequency and type stability.
  • Preserve missing fields, explicit nulls, mixed types, nested objects, arrays, and numeric values. Unsupported extraction shapes remain in the residual without data loss.
  • Store extracted values once in the current record representation. Indexes, WAL/history, and replicas still have their normal storage costs.
  • Persist layout and policy metadata. Reopening must not infer a different interpretation of existing data.
  • Automatically discover additional fields on later writes. Queries and full-document reconstruction must remain correct while older rows retain values in residual storage.
  • Keep data visibility atomic at the record level and reject incompatible/corrupt layout metadata.
  • Support text indexing/search and logical field access without requiring callers to know generated physical column names.
  • Validate real storage round trips, schema evolution, mixed-type fallbacks, large text, reopen, projections, and indexed retrieval. Benchmark bytes, I/O, latency, and memory before claiming performance gains.

Incremental delivery

  • Automatic bounded profiling, deterministic layout selection, and lossless shred/reconstruct codec with semantic tests. PR: #833.
  • Persistent DocumentStore with automatic schema evolution, point reads, projections, and reopen/version tests using existing Lance primitives. PR: #835.
  • Logical field filtering and full-text indexing/search over automatically selected text columns, including residual fallbacks and historical backfill. PR: #836.
  • Public Python API, executable examples, and user documentation. PR: #839.
  • Reproducible equivalent-release workload benchmarks and measured I/O/compression fixes, with raw results and explicit preparation/history/memory costs. PRs: #869, #871.
  • Background re-shredding and query-aware index/selection policy; extend array-path extraction based on evidence.
  • Coordinated WAL/HTTP integration for the embedded document API.

Each increment should be a small English-language PR linked here. Merge only after its CI checks pass. This issue remains open until the tracked work is complete; an initial increment is not equivalent to completion of the full roadmap.

Scope boundaries

The initial implementation restores JSON values and structure, not the exact original whitespace or object-key order. Automatic selection is required from the first increment; automatic indexing and background rewriting are separate policies. Standard Arrow Variant interoperability is optional, not a prerequisite for implementing automatic shredding in lance-context.

Initial implementation delivered

PRs #833, #835, #836, and #839 are merged. Every PR passed its GitHub CI before merge. Automatic selection, schema evolution, lossless reconstruction, logical filtering, historical backfill, automatic full-text indexing, and the public Python API use existing Lance primitives with no Lance source changes.

Validation includes the full local workspace checks, 16 new Rust tests, eight Python document-store cases, and a 20 MiB text-field round trip with projected reads, search, and reopen on Lance 14. The local Python suite passed with 228 passed, 11 skipped, and 1 expected failure. PR #839 passed all 18 GitHub checks.

The delivered API is embedded, with one writer per dataset. Search prepares historical text/indexes synchronously. Background maintenance, coordinated WAL/HTTP integration, query-aware policies and array-path extraction remain follow-up work; this parent stays open for the roadmap above. Workload benchmarks are delivered in the acceptance increment below.

Performance acceptance delivered (2026-11-13)

PR Merge commit Merge time (UTC) Passing checks
https://github.com/lance-format/lance-context/pull/869 2fe2a5e6891d3e8dbb8e37979e9c1e241835de58 2026-11-13T05:39:01Z 15
https://github.com/lance-format/lance-context/pull/871 56b81d156dfd5cb227103c6305aba7d31654c538 2026-11-13T06:23:53Z 18

Full fixed-code local validation passed: 1,342 Rust tests passed (416 ignored), Python 228 passed / 11 skipped / 1 expected failure, workspace formatting/lint, Ruff and Pyright. Python used a freshly rebuilt fixed native extension. Feature-enabled regression/example lint and all three release I/O/storage regressions passed. The added semantic case covers 1,030 documents, container/root/scalar-ancestor queries, null/missing, batching, backfill and pinned snapshots. Both I/O and storage regressions were first demonstrated failing on the old implementation.

All 54 equivalent-release benchmark runs passed assertions. For 8 × 20 MiB records, residual-filter median falls from 247.086 ms to 0.430 ms; requested filter bytes fall from 39,797,813 to 720. Current data after maintenance is 37.957 MiB versus 75.910 MiB for JSON plus duplicated text. Full reads take 28.312 ms versus 31.857 ms. These are local synthetic warm-cache results, not production capacity or cold-disk guarantees. The baseline stores serialized JSON in LargeUtf8, not JSONB.

Costs remain: initial FTS preparation is synchronous and takes 3.356 s versus 1.764 s; retaining history brings total directory size to 114.272 MiB versus 76.107 MiB. Peak RSS remains higher than the duplicated baseline, and medium-case RSS increased with batching. Current-record savings do not imply lower total retained storage or faster performance for every operation.

The initial feature needed no Lance source changes. Performance acceptance adds a minimal 13-line guard in the vendored Lance encoder: single variable-width values larger than 32 KiB use existing compression instead of raw constant pages. It covers both offset widths of UTF-8 and binary arrays; multi-value constants and the on-disk format are unchanged. No upstream Lance PR was needed.

Performance report and all raw samples. Both PRs were merged only after every GitHub check passed.

Ngôn ngữ chính
Rust
Star
89
Fork
21
Merge trung bình
2 giờ 6 phút
Pull request đã merge (30 ngày)
435

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của lance-format/lance-context

Tất cả issue của lance-format/lance-context

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.