feat: automatic JSON shredding and transparent document access
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 20/100
- issue の種類
- 機能追加
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
調査の方向性
Read the remaining roadmap item in this issue: background re-shredding, workload benchmarks, and query-aware index and selection policy; array-path extraction is explicitly evidence-driven. Start by reviewing the merged PRs #833, #835, #836, and #839 to understand the current DocumentStore and Lance primitives. Define a bounded follow-up increment with tests and benchmarks; it is done when that increment's behavior is validated and its English-language PR passes CI.
索引モデルが issue の本文から書いたものです。
説明
Problem
Large JSON documents often contain one or two text fields that dominate their size. Keeping a complete JSON document plus extracted text columns duplicates those values. Users should not have to declare extraction paths, manage residual documents, or reconstruct records themselves.
Goal
Implement real automatic JSON shredding in lance-context: discover useful fields from incoming documents, choose typed physical columns automatically, remove extracted values from the residual, and expose the original logical document through reads and queries. Keep policy, reconstruction, and layout evolution in lance-context; use existing Lance storage/indexing primitives and avoid upstream changes unless a demonstrated gap requires a small fix.
Design requirements
- No user-maintained list of shredded paths. Bound profiling, column growth, and memory use; account for value bytes as well as frequency and type stability.
- Preserve missing fields, explicit nulls, mixed types, nested objects, arrays, and numeric values. Unsupported extraction shapes remain in the residual without data loss.
- Store extracted values once in the current record representation. Indexes, WAL/history, and replicas still have their normal storage costs.
- Persist layout and policy metadata. Reopening must not infer a different interpretation of existing data.
- Automatically discover additional fields on later writes. Queries and full-document reconstruction must remain correct while older rows retain values in residual storage.
- Keep data visibility atomic at the record level and reject incompatible/corrupt layout metadata.
- Support text indexing/search and logical field access without requiring callers to know generated physical column names.
- Validate real storage round trips, schema evolution, mixed-type fallbacks, large text, reopen, projections, and indexed retrieval. Benchmark bytes, I/O, latency, and memory before claiming performance gains.
Incremental delivery
- Automatic bounded profiling, deterministic layout selection, and lossless shred/reconstruct codec with semantic tests. PR: #833.
- Persistent DocumentStore with automatic schema evolution, point reads, projections, and reopen/version tests using existing Lance primitives. PR: #835.
- Logical field filtering and full-text indexing/search over automatically selected text columns, including residual fallbacks and historical backfill. PR: #836.
- Public Python API, executable examples, and user documentation. PR: #839.
- Reproducible equivalent-release workload benchmarks and measured I/O/compression fixes, with raw results and explicit preparation/history/memory costs. PRs: #869, #871.
- Background re-shredding and query-aware index/selection policy; extend array-path extraction based on evidence.
- Coordinated WAL/HTTP integration for the embedded document API.
Each increment should be a small English-language PR linked here. Merge only after its CI checks pass. This issue remains open until the tracked work is complete; an initial increment is not equivalent to completion of the full roadmap.
Scope boundaries
The initial implementation restores JSON values and structure, not the exact original whitespace or object-key order. Automatic selection is required from the first increment; automatic indexing and background rewriting are separate policies. Standard Arrow Variant interoperability is optional, not a prerequisite for implementing automatic shredding in lance-context.
Initial implementation delivered
PRs #833, #835, #836, and #839 are merged. Every PR passed its GitHub CI before merge. Automatic selection, schema evolution, lossless reconstruction, logical filtering, historical backfill, automatic full-text indexing, and the public Python API use existing Lance primitives with no Lance source changes.
Validation includes the full local workspace checks, 16 new Rust tests, eight Python document-store cases, and a 20 MiB text-field round trip with projected reads, search, and reopen on Lance 14. The local Python suite passed with 228 passed, 11 skipped, and 1 expected failure. PR #839 passed all 18 GitHub checks.
The delivered API is embedded, with one writer per dataset. Search prepares historical text/indexes synchronously. Background maintenance, coordinated WAL/HTTP integration, query-aware policies and array-path extraction remain follow-up work; this parent stays open for the roadmap above. Workload benchmarks are delivered in the acceptance increment below.
Performance acceptance delivered (2026-11-13)
| PR | Merge commit | Merge time (UTC) | Passing checks |
|---|---|---|---|
| https://github.com/lance-format/lance-context/pull/869 | 2fe2a5e6891d3e8dbb8e37979e9c1e241835de58 | 2026-11-13T05:39:01Z | 15 |
| https://github.com/lance-format/lance-context/pull/871 | 56b81d156dfd5cb227103c6305aba7d31654c538 | 2026-11-13T06:23:53Z | 18 |
Full fixed-code local validation passed: 1,342 Rust tests passed (416 ignored), Python 228 passed / 11 skipped / 1 expected failure, workspace formatting/lint, Ruff and Pyright. Python used a freshly rebuilt fixed native extension. Feature-enabled regression/example lint and all three release I/O/storage regressions passed. The added semantic case covers 1,030 documents, container/root/scalar-ancestor queries, null/missing, batching, backfill and pinned snapshots. Both I/O and storage regressions were first demonstrated failing on the old implementation.
All 54 equivalent-release benchmark runs passed assertions. For 8 × 20 MiB records, residual-filter median falls from 247.086 ms to 0.430 ms; requested filter bytes fall from 39,797,813 to 720. Current data after maintenance is 37.957 MiB versus 75.910 MiB for JSON plus duplicated text. Full reads take 28.312 ms versus 31.857 ms. These are local synthetic warm-cache results, not production capacity or cold-disk guarantees. The baseline stores serialized JSON in LargeUtf8, not JSONB.
Costs remain: initial FTS preparation is synchronous and takes 3.356 s versus 1.764 s; retaining history brings total directory size to 114.272 MiB versus 76.107 MiB. Peak RSS remains higher than the duplicated baseline, and medium-case RSS increased with batching. Current-record savings do not imply lower total retained storage or faster performance for every operation.
The initial feature needed no Lance source changes. Performance acceptance adds a minimal 13-line guard in the vendored Lance encoder: single variable-width values larger than 32 KiB use existing compression instead of raw constant pages. It covers both offset widths of UTF-8 and binary arrays; multi-value constants and the on-disk format are unchanged. No upstream Lance PR was needed.
Performance report and all raw samples. Both PRs were merged only after every GitHub check passed.
- 主要言語
- Rust
- スター
- 89
- フォーク
- 21
- 平均マージ
- 2時間 6分
- マージ済み PR(30日)
- 435
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
lance-format/lance-context のほかの issue
-
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
lance-format/lance-context#378 · コメント 119 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 6/100
lance-format/lance-context#333 · コメント 11 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 30/100
lance-format/lance-context#260 ·
メンテナーはふだん 1 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 28/100
lance-format/lance-context#214 ·
メンテナーはふだん 1 日以内に返信
-
RolloutStore::dataset should be interior-mutable so merge/compact can refresh the handle without an exclusive lock対応中かも @lucyge2022 が 97 日前に担当しました。 オープンenhancement
難易度 5/5 1週間以上 初心者へのやさしさ 42/100
lance-format/lance-context#198 ·
メンテナーはふだん 1 日以内に返信
lance-format/lance-context の issue をすべて見る
似ている issue
-
area:cli area:security bug priority:medium
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
status: new topic: ga4gh-sdk type: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
メンテナーはふだん 1 日以内に返信
-
comp-rbac
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
ClickHouse/ClickHouse#131875 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
googleworkspace/cli#984 ·
メンテナーはふだん 1 日以内に返信