Consolidate the storage layer of the three fixed schemas (rollout as reference) + support arbitrary user-defined schemas
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức phù hợp với người mới
- 28/100
Hướng nghiên cứu
Bắt đầu bằng cách đọc các ticket con được đề xuất và các phần triển khai hiện có trong crates/lance-context-core/src/rollout_store.rs, store.rs, datagen_store.rs và registry.rs. So sánh hành vi lưu trữ rollout với các store khác, sau đó chọn một ticket con có phạm vi rõ ràng; kết quả hoàn tất phải khớp với hành vi được nêu trong ticket đó mà không âm thầm thay đổi các hợp đồng của API hoặc schema.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Summary
lance-context-core currently ships three independently-written fixed-schema stores:
| Store | File | Key column | Schema |
|---|---|---|---|
RolloutStore |
crates/lance-context-core/src/rollout_store.rs |
id |
rollout_schema() (36 cols) |
ContextStore |
crates/lance-context-core/src/store.rs |
id |
schema_with_options() (26 cols) |
DatagenStore |
crates/lance-context-core/src/datagen_store.rs |
event_id |
datagen_log_schema() (33 cols) |
They are structurally parallel but share almost no code. Over time they have forked at the storage layer, and their observable behavior now differs in ways that are bugs, not design choices.
This is a parent ticket covering two related goals:
- Consolidate the low-level storage layer of the three stores onto one shared implementation, taking
RolloutStoreas the reference/canonical behavior. - Support arbitrary user-defined schemas on top of that shared layer, with a small set of enforced constraints (notably a required
idfield), while keepingaddandmergesemantics identical to the fixed-schema stores.
Part 1 — Consolidate the storage layer (rollout store is the reference)
Divergences we need to fix
Write / add path
RolloutStoreandDatagenStorekeep a residentShardWriterand doput→force_seal_active→wait_for_flush_drainon every add, so a generation is durable and cross-instance-visible whenaddreturns.ContextStoreopens and closes a new writer per call (write_entries), with no seal and no drain. Its writes are only visible after an internal Lance flush. It also has no fence-retry handling, because it never holds a writer.- Result:
addmeans three different things depending on which store you call.
WAL -> base merge
RolloutStore::merge_own_shardis the correct implementation: close writer, scan generations,WriteMode::Append,claim_epoch+commit_updatewith a surgical generation drain, then remove the shard dir. Triggered by both a count threshold (maybe_merge_own_shard) and a time threshold (cleanup_own_shard).DatagenStore::merge_own_shardis a near line-for-line copy (plus a background cleanup task), and buffers every flushed generation fully in memory.ContextStorenever merges at all. WAL generations accumulate forever and every read pays a fullLsmScannerreconstruction over all shard manifests.
Compaction
RolloutStore: synchronous, single-compactor,defer_index_remap: trueto work around a Lance panic on the fieldless MemWAL index.ContextStore: has compaction plus a background task guarded byCompactionState, but withoutdefer_index_remap.DatagenStore: no compaction at all — base fragments grow unbounded.
Indexing
- Rollout: manual ZoneMap on
id. - Context: automatic ZoneMap-or-BTree on open and post-compaction, but reopens with bare
Dataset::open(uri)— droppingstorage_options, which breaks on credentialed object stores. No vector index is ever built; search is a brute-force in-memory scan. - Datagen: none.
Open / create
- The
load_with_options/create_with_options/DatasetNotFound-fallback triad is written out four times (three stores +registry.rs), and theWriteParams+ObjectStoreParams+StorageOptionsAccessoridiom appears ~6 times.
Read path
ContextStore::getscansself.dataset.scan()directly, bypassing the LSM scanner, so it can miss unflushed WAL rows — inconsistent with its ownget_by_id.get_by_id/get_by_external_iddo a fulllist()+ linearfind()instead of using the id index.ContextStoretruncates limit/offset in memory instead of pushing down.- Only
RolloutStore's decode path tolerates missing columns (column_as_optional).
Errors
- Three conventions in-tree: inline
LanceError::from(ArrowError::...)(rollout), a localfn invalid_input(datagen), nativeLanceError::invalid_input(eval), plusResult<_, String>inrecord.rsand the filter parsers. api_impl.rs::to_ctx_errstring-matchesDisplayoutput to recover a typed error, contradicting the stated intent of re-exportinglance::Errorfromlib.rs.is_fenced_erroris likewise string-matching and duplicated in two files.
Misc
update_visible_recordmintsUuid::new_v4()even thoughid.rsdocuments UUIDv7 for index-tail clustering.RolloutFilters::from_json_valueallows 6 keys while its ownexpression()supports 9 —datasetandcontent_typeare unreachable from JSON.- Nested structs (
relationships,state_metadata) are decoded positionally; field reordering silently corrupts data.
Target
A single shared storage layer (working name lance-context-core::store_base) owning:
- open / create /
load_with_options/create_with_options/ storage-options plumbing - resident writer lifecycle, fence detection + retry,
Drop/close() add: encode ->ensure_mem_wal->put->force_seal_active->wait_for_flush_drain(rollout semantics, for all stores)- WAL merge: one
merge_own_shard, count + time triggers, surgical generation drain, streaming (not fully buffered) - compaction with
defer_index_remap, single-compactor guard, optional background task - id index management (ensure-on-open, re-ensure post-compaction), always via
load_with_options - LSM scanner construction with a configurable merge key
- one error taxonomy, with typed fence/not-found detection instead of string matching
RolloutStore, ContextStore, DatagenStore become thin schema+encode/decode layers over it. The features unique to ContextStore (lifecycle, supersession, tombstones, id-uniqueness validation) move up into the shared layer as opt-in capabilities rather than being dropped.
Existing seams to build on: crate::store's column_as / column_as_optional / timestamp_from_micros / CompactionConfig / CompactionStats, rollout_store::derive_shard_id, crate::storage (join_uri, validate_store_name), crate::id::generate_id.
Behavior changes to call out explicitly: ContextStore.add becomes synchronously durable and visible; ContextStore starts merging WAL into the base table; DatagenStore gains compaction and an event_id index. These are the point of the ticket, but they are visible changes and need release notes.
Part 2 — Arbitrary user-defined schemas
Users should be able to declare their own schema at store-creation time instead of picking one of three baked-in ones — but the storage behavior underneath must stay identical, especially add and merge.
Constraints to enforce at creation
- An
idcolumn is mandatory:Utf8, non-nullable, taggedlance-schema:unenforced-primary-key=true. It is always the LSM merge key. - Reserved / prefixed names rejected:
_rowid,_mem_wal, and the tombstonecontent_typesentinel. - Only Arrow types supported by the generic encode/decode path.
- Blob columns declared explicitly (today
VALID_BLOB_COLUMNSis a hardcoded 2-element list). - Vector columns declared with dimension + distance metric so an index can actually be built (today this is schema metadata
lance-context:distance_metric). - Schema is immutable after creation in v1; evolution is a follow-up (only
migrate_relationships_columnexists today).
Why this is currently hard
Each store spells out its field list in five to seven independent places: schema fn, record->Arrow builders, Arrow->record decode, projection list, filter parser, merge key, index name. DatagenStore::events_to_batch is positional and unchecked.
The one genuinely schema-driven piece already in-tree is the final assembly step of both records_to_batch implementations, which iterates the dataset's fields and pulls from a name->array map, erroring on unknown columns. That already handles column reordering and omission — it just doesn't handle addition. That is the seed of the generic path.
ContextStore::schema_with_options is the only schema that is built rather than declared constant, so it is the natural insertion point.
API surface work
There is currently no schema abstraction at the API boundary at all — lance-context-api, -server and -client import zero arrow types, and transport is hand-written REST/JSON over axum with no proto/OpenAPI/codegen.
CreateContextRequest/CreateRolloutStoreRequestmust carry a schema declaration, andregistry.rsmust persist it (today it stores onlyname/uri/created_at).AddRecordRequest(20 fields, and it has noid— the server generates one),AddRolloutRequest(44 fields, does take a clientid, so it is the better template),RecordDto,RolloutRecordDto,RecordPatchDto+ its hand-written 16-wayis_empty()all become schema-driven.- Six duplicated conversion fns need deduping first:
record_from_add_requestandrollout_record_from_add_requesteach exist in bothroutes/andapi_impl.rs; server handlers bypass the API traits and call the inherentadddirectly. SearchRequest/RetrieveRequestneed explicit column names once more than one vector/text column can exist.RolloutStoreApi::get_trajectory's default body hardcodesrollout_id/sequence_orderinside the trait — must move out.- Client
add_rolloutsselects binary parts bybinary_payload.is_some(); must become schema-driven. - Validation is scattered across four layers with no choke point, and no
deny_unknown_fieldsanywhere (unknown keys are silently dropped). Needs one schema-driven validation entry point. DatagenStorehas no API/server/client surface at all — decide whether it gets one or stays internal.
specs/rollout-schema-design.md explicitly documents the current one-schema-at-a-time-by-hand approach and should be superseded.
Proposed child tickets
- Extract shared open/create/storage-options module; collapse the 4 duplicated triads.
- Extract resident writer + fence-retry +
close/Drop; migrateContextStoreonto it. - Unify the
addpath on rollout semantics (put-> seal -> drain); migrateContextStoreandDatagenStore. - Extract one
merge_own_shard+ count/time triggers; make it streaming; wire upContextStore(new behavior). - Unify compaction:
defer_index_remapeverywhere, single-compactor guard, opt-in background task; add compaction toDatagenStore. - Unify index management; fix the
Dataset::openstorage-options drop; add anevent_idindex toDatagenStore. - Unify the error taxonomy; delete
to_ctx_errstring-matching and the duplicatedis_fenced_error. - Read-path fixes: LSM-based
ContextStore::get, index-accelerated point lookups, pushdown of limit/offset. - Name-based (not positional) nested struct encode/decode.
- Generic schema-driven encode/decode engine (name-keyed builders + decoders).
SchemaSpectype + validation rules (requiredid, reserved names, blob/vector declarations) + registry persistence.- API/server/client: schema-carrying create, generic record DTOs, dedupe the 6 conversion fns, explicit column names in search/retrieve.
- Docs: supersede
specs/rollout-schema-design.md, write the arbitrary-schema guide + migration/release notes.
Out of scope
- Schema evolution after creation (follow-up).
- Changing the wire protocol away from REST/JSON.
- Vector index creation for
ContextStoresearch (tracked separately, though the shared index layer should make room for it).
- Ngôn ngữ chính
- Rust
- Star
- 89
- Fork
- 21
- Merge trung bình
- 2 giờ 6 phút
- Pull request đã merge (30 ngày)
- 435
Chuẩn bị môi trường
Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của lance-format/lance-context
-
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 20/100
lance-format/lance-context#832 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 35/100
lance-format/lance-context#378 · 119 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 6/100
lance-format/lance-context#333 · 11 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 30/100
lance-format/lance-context#260 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
RolloutStore::dataset should be interior-mutable so merge/compact can refresh the handle without an exclusive lockCó thể đã có người làm @lucyge2022 đã nhận 97 ngày trước. Đang mởenhancement
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 42/100
lance-format/lance-context#198 ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của lance-format/lance-context
Issue tương tự
-
area:cli area:security bug priority:medium
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
rtk-ai/rtk#5127 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
status: new topic: ga4gh-sdk type: bug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 75/100
Maintainer thường phản hồi trong vòng 1 ngày
-
comp-rbac
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
ClickHouse/ClickHouse#131875 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 70/100
Maintainer thường phản hồi trong vòng 1 ngày
-
`gws auth login` keeps the old token cache, so API calls use a token from the previous grantĐang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 80/100
googleworkspace/cli#984 ·
Maintainer thường phản hồi trong vòng 1 ngày