feat: automatic JSON shredding and transparent document access
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 20/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Ambito
- data-engineering, databases
Direzione di ricerca
Read the remaining roadmap item in this issue: background re-shredding, workload benchmarks, and query-aware index and selection policy; array-path extraction is explicitly evidence-driven. Start by reviewing the merged PRs #833, #835, #836, and #839 to understand the current DocumentStore and Lance primitives. Define a bounded follow-up increment with tests and benchmarks; it is done when that increment's behavior is validated and its English-language PR passes CI.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem
Large JSON documents often contain one or two text fields that dominate their size. Keeping a complete JSON document plus extracted text columns duplicates those values. Users should not have to declare extraction paths, manage residual documents, or reconstruct records themselves.
Goal
Implement real automatic JSON shredding in lance-context: discover useful fields from incoming documents, choose typed physical columns automatically, remove extracted values from the residual, and expose the original logical document through reads and queries. Keep policy, reconstruction, and layout evolution in lance-context; use existing Lance storage/indexing primitives and avoid upstream changes unless a demonstrated gap requires a small fix.
Design requirements
- No user-maintained list of shredded paths. Bound profiling, column growth, and memory use; account for value bytes as well as frequency and type stability.
- Preserve missing fields, explicit nulls, mixed types, nested objects, arrays, and numeric values. Unsupported extraction shapes remain in the residual without data loss.
- Store extracted values once in the current record representation. Indexes, WAL/history, and replicas still have their normal storage costs.
- Persist layout and policy metadata. Reopening must not infer a different interpretation of existing data.
- Automatically discover additional fields on later writes. Queries and full-document reconstruction must remain correct while older rows retain values in residual storage.
- Keep data visibility atomic at the record level and reject incompatible/corrupt layout metadata.
- Support text indexing/search and logical field access without requiring callers to know generated physical column names.
- Validate real storage round trips, schema evolution, mixed-type fallbacks, large text, reopen, projections, and indexed retrieval. Benchmark bytes, I/O, latency, and memory before claiming performance gains.
Incremental delivery
- Automatic bounded profiling, deterministic layout selection, and lossless shred/reconstruct codec with semantic tests. PR: #833.
- Persistent DocumentStore with automatic schema evolution, point reads, projections, and reopen/version tests using existing Lance primitives. PR: #835.
- Logical field filtering and full-text indexing/search over automatically selected text columns, including residual fallbacks and historical backfill. PR: #836.
- Public Python API, executable examples, and user documentation. PR: #839.
- Reproducible equivalent-release workload benchmarks and measured I/O/compression fixes, with raw results and explicit preparation/history/memory costs. PRs: #869, #871.
- Background re-shredding and query-aware index/selection policy; extend array-path extraction based on evidence.
- Coordinated WAL/HTTP integration for the embedded document API.
Each increment should be a small English-language PR linked here. Merge only after its CI checks pass. This issue remains open until the tracked work is complete; an initial increment is not equivalent to completion of the full roadmap.
Scope boundaries
The initial implementation restores JSON values and structure, not the exact original whitespace or object-key order. Automatic selection is required from the first increment; automatic indexing and background rewriting are separate policies. Standard Arrow Variant interoperability is optional, not a prerequisite for implementing automatic shredding in lance-context.
Initial implementation delivered
PRs #833, #835, #836, and #839 are merged. Every PR passed its GitHub CI before merge. Automatic selection, schema evolution, lossless reconstruction, logical filtering, historical backfill, automatic full-text indexing, and the public Python API use existing Lance primitives with no Lance source changes.
Validation includes the full local workspace checks, 16 new Rust tests, eight Python document-store cases, and a 20 MiB text-field round trip with projected reads, search, and reopen on Lance 14. The local Python suite passed with 228 passed, 11 skipped, and 1 expected failure. PR #839 passed all 18 GitHub checks.
The delivered API is embedded, with one writer per dataset. Search prepares historical text/indexes synchronously. Background maintenance, coordinated WAL/HTTP integration, query-aware policies and array-path extraction remain follow-up work; this parent stays open for the roadmap above. Workload benchmarks are delivered in the acceptance increment below.
Performance acceptance delivered (2026-11-13)
| PR | Merge commit | Merge time (UTC) | Passing checks |
|---|---|---|---|
| https://github.com/lance-format/lance-context/pull/869 | 2fe2a5e6891d3e8dbb8e37979e9c1e241835de58 | 2026-11-13T05:39:01Z | 15 |
| https://github.com/lance-format/lance-context/pull/871 | 56b81d156dfd5cb227103c6305aba7d31654c538 | 2026-11-13T06:23:53Z | 18 |
Full fixed-code local validation passed: 1,342 Rust tests passed (416 ignored), Python 228 passed / 11 skipped / 1 expected failure, workspace formatting/lint, Ruff and Pyright. Python used a freshly rebuilt fixed native extension. Feature-enabled regression/example lint and all three release I/O/storage regressions passed. The added semantic case covers 1,030 documents, container/root/scalar-ancestor queries, null/missing, batching, backfill and pinned snapshots. Both I/O and storage regressions were first demonstrated failing on the old implementation.
All 54 equivalent-release benchmark runs passed assertions. For 8 × 20 MiB records, residual-filter median falls from 247.086 ms to 0.430 ms; requested filter bytes fall from 39,797,813 to 720. Current data after maintenance is 37.957 MiB versus 75.910 MiB for JSON plus duplicated text. Full reads take 28.312 ms versus 31.857 ms. These are local synthetic warm-cache results, not production capacity or cold-disk guarantees. The baseline stores serialized JSON in LargeUtf8, not JSONB.
Costs remain: initial FTS preparation is synchronous and takes 3.356 s versus 1.764 s; retaining history brings total directory size to 114.272 MiB versus 76.107 MiB. Peak RSS remains higher than the duplicated baseline, and medium-case RSS increased with batching. Current-record savings do not imply lower total retained storage or faster performance for every operation.
The initial feature needed no Lance source changes. Performance acceptance adds a minimal 13-line guard in the vendored Lance encoder: single variable-width values larger than 32 KiB use existing compression instead of raw constant pages. It covers both offset widths of UTF-8 and binary arrays; multi-value constants and the on-disk format are unchanged. No upstream Lance PR was needed.
Performance report and all raw samples. Both PRs were merged only after every GitHub check passed.
- Lingua principale
- Rust
- Stelle
- 89
- Fork
- 21
- Merge medio
- 2h 6m
- PR unite (30g)
- 435
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di lance-format/lance-context
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
lance-format/lance-context#378 · 119 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 6/100
lance-format/lance-context#333 · 11 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
lance-format/lance-context#260 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 28/100
lance-format/lance-context#214 ·
I maintainer di solito rispondono entro 1 giorno
-
RolloutStore::dataset should be interior-mutable so merge/compact can refresh the handle without an exclusive lockForse già presa @lucyge2022 l’ha presa 97 giorni fa. Apertaenhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 42/100
lance-format/lance-context#198 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di lance-format/lance-context
Issue simili
-
comp-rbac
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ClickHouse/ClickHouse#131875 ·
I maintainer di solito rispondono entro 1 giorno
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesForse già presa @Borg93 l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#11117 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Forse già presa @Ynyesto l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
foundry-rs/foundry#18485 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
bondybondbond/rushcut#238 · 1 commento ·