Tracking Issue: Architecture, Design Invariants, and Roadmap for Official lance-go

Aperta
#1 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github, go
Ambito
backend

Direzione di ricerca

This is a broad architecture and roadmap tracker, and the body reports substantial implementation already merged. Start by checking the linked roadmap pull requests and the remaining unchecked work, then read the relevant entry points named in the repository layout, such as dataset.go, internal/ffi, and namespace/. Done means completing a specific open roadmap item and its stated acceptance checks; this issue alone does not define a bounded first contribution.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Summary

This issue tracks the architecture, design invariants, upstream dependencies, and phased implementation roadmap for lance-go (github.com/lance-format/lance-go) — the official Go SDK for the Lance columnar data format (see community proposal lance-format/lance#7262).

Our goal is to provide first-class API parity with Rust lance, pylance, and lance-java while remaining 100% CGO_ENABLED=0 compatible, idiomatic to modern Go (Go 1.23+ iterators, context.Context cancellation, io interfaces, concurrency-safe immutable snapshots), and zero-configuration out of the box.


Core Architectural Decisions

  1. CGO_ENABLED=0 FFI via ebitengine/purego + Official lance-c

    • Instead of maintaining a separate Rust FFI shim crate in this repository, lance-go binds directly to the official lance-format/lance-c shared library (liblance_c.{so,dylib,dll}).
    • The currently bundled ac2634c header declares 878 exported C entry points using opaque pointers, scalars/enums, out-pointers, and function pointers (0 struct-by-value arguments or return types). Pure-Go ebitengine/purego supports the entire ABI across Linux, macOS, and Windows (amd64 & arm64) without requiring CGo or runtime libffi.
    • lance-go releases track lance-c versions directly; missing C ABI capabilities are contributed upstream to lance-c.
  2. Pure-Go Arrow C Data Interface Bridge (internal/cdata)

    • github.com/apache/arrow-go/v18 supports CGO_ENABLED=0, except arrow/cdata which is gated behind //go:build cgo.
    • lance-go implements a pure-Go Arrow C Data Interface (ArrowSchema, ArrowArray, ArrowArrayStream) for zero-copy bidirectional arrow.RecordBatch / array.RecordReader exchange.
    • Because CGO_ENABLED=0 avoids CGo pointer-passing restrictions, exported Go-allocated Arrow buffers (memory.GoAllocator) are retained and pinned via runtime.Pinner until the C ArrowArray.release callback fires — enabling zero-copy writes without forcing callers onto a custom C malloc allocator.
  3. Zero-Config Out-of-the-Box Distribution + Enterprise Build Escape Hatches

    • By default (!lance_external_lib), platform-constrained files (GOOS/GOARCH) blank-import a single per-platform companion Go module (github.com/lance-format/lance-go/artifacts/<os>-<arch>) containing a stripped, zstd-compressed liblance_c shared library.
    • Standard users only run go get github.com/lance-format/lance-go and CGO_ENABLED=0 go test ./... — Ordinary target builds are intended to fetch only the current target platform's artifact archive (subject to cold-cache consumer validation) and extract it once into a content-addressed cache directory (<UserCacheDir>/lance-go/<sha256>/) guarded by file locks.
    • Custom/container/hermetic builds can pass -tags lance_external_lib (excluding bundled artifact packages from compilation and the resulting binary; module metadata and the module graph may still reference artifact modules) and/or set LANCE_LIB_PATH to load a system-provided liblance_c. Commands that inspect all platforms or dependencies, including go mod tidy, may download additional artifact archives; external mode does not promise zero downloads. The default retains a single import path and requires no separate bootstrap module.
  4. Strict Immutable MVCC Snapshot Semantics for *lance.Dataset

    • Go services routinely share a *lance.Dataset across hundreds of reader goroutines.
    • Every mutating, schema-evolution, indexing, or maintenance operation clones the underlying dataset handle in O(1) (lance_dataset_clone_handle), executes the mutation on the cloned handle, and returns a new *lance.Dataset pointing to the newly committed version — leaving the receiver *Dataset completely untouched and lock-free for concurrent readers.
  5. lance-namespace Alignment with pylance & lance-java

    • Expose an idiomatic pure-Go namespace.Namespace interface and encoding/json OpenAPI models on top of lance-c's native LanceNamespace bindings (DirectoryNamespace, RestNamespace, RestAdapter, custom Go callback adapters, dynamic context providers, automatic credential refresh, and namespace-managed commits).

Repository & Multi-Module Layout

lance-go/
├── go.mod                          # module github.com/lance-format/lance-go (pure Go)
├── dataset.go                      # Dataset lifecycle, read-only metadata, version/branch/tag checkout
├── dataset_write.go                # Write, Append, Delete, Update, MergeInsert, Merge (returns new *Dataset)
├── dataset_schema.go               # AddColumns*, AlterColumns, DropColumns, Metadata & Config updates
├── scanner.go                      # Scanner options, Records(ctx) iter.Seq2, ToArrowReader, Take
├── fragment.go                     # Fragment metadata inspection, uncommitted WriteFragments
├── transaction.go                  # Transaction builder/inspector, Protobuf serde, CommitTransaction
├── blob.go                         # TakeBlobs, BlobFile (io.Reader, io.Seeker, io.ReaderAt, io.Closer)
├── session.go                      # Session caches/stats, StorageOptionsProvider callback bridge
├── errors.go                       # LanceErrorCode mapping & sentinel errors (errors.Is)
├── options.go                      # Functional options (OpenOption, WriteOption, ScanOption, ...)
├── lib_auto_<os>_<arch>.go         # //go:build !lance_external_lib && <os> && <arch>
│
├── file/                           # package file: Raw .lance v2 file Reader / Writer
├── index/                          # package index: Vector, scalar & FTS index params + distributed SegmentBuilder
├── compaction/                     # package compaction: Compaction V2 (Plan/Execute/Commit) & OptimizeIndices
├── namespace/                      # package namespace: Namespace interface, models, Directory/Rest/Adapter
│
├── internal/
│   ├── loader/                     # Library discovery, zstd content-addressed cache + flock, dlopen
│   ├── handle/                     # Sharded lock-free/concurrent uintptr <-> Go object handle table
│   ├── ffi/                        # purego symbol bindings, static callback trampolines, poll scheduler
│   └── cdata/                      # Pure-Go Arrow C Data Interface (Schema/Array/Stream + runtime.Pinner)
│
├── artifacts/                      # Separate Go submodules isolating prebuilt stripped zstd shared libs
│   ├── linux-amd64/go.mod
│   ├── linux-arm64/go.mod
│   ├── darwin-arm64/go.mod
│   └── windows-amd64/go.mod
│
└── scripts/
    ├── sync-lance-c.sh             # Fetch/verify/recompress lance-c release artifacts
    └── check-abi-symbols.go        # Verify internal/ffi registrations against lance.h

Hard Engineering Invariants

  1. Thread-Local Error Ordering & OS Thread Pinning

    • lance-c stores errors in Rust thread-local storage (LAST_ERROR), and lance_last_error_message() calls .take(), clearing the stored error code.
    • Every FFI call that may inspect lance_last_error_* must run inside runtime.LockOSThread() / defer runtime.UnlockOSThread(), and must call lance_last_error_code() before lance_last_error_message(), followed by lance_free_string().
  2. Global Static Callback Trampolines (purego.NewCallback 2,000-Slot Limit)

    • purego has a fixed process-wide limit of 2,000 callback slots that are never garbage-collected.
    • internal/ffi must declare all C function-pointer parameters as uintptr, allocate a fixed set (< 15) of global static trampolines once via sync.Once, pass per-call state as integer handles via internal/handle, and recover any Go panics before crossing the C ABI boundary.
  3. Explicit Resource Lifetimes & Zero-Copy Memory Safety

    • Imported C ArrowArray buffers share a batch-level reference count and invoke C release deterministically when Release() reaches zero (never relying on runtime.SetFinalizer as the primary reclamation mechanism for native buffers).
    • Exported Go arrow.RecordBatch buffers retain the batch and pin struct + pointer arrays + data buffers via runtime.Pinner until Rust invokes cbReleaseArray.
  4. Goroutine-Friendly Async poll_next Scanning & Teardown Ordering

    • Scanner.Records(ctx) drives lance_scanner_poll_next with LanceWaker so blocked I/O parks only the Go goroutine (G) without pinning OS threads (M) and supports immediate ctx.Done() cancellation.
    • Never call lance_scanner_close inside a LanceWaker callback; always wait for lance_scanner_close to return before releasing the waker handle from internal/handle.
  5. Active-Operation Reference Counting on *Dataset

    • Guard native *LanceDataset pointers with an in-flight operation refcount so concurrent ds.Close() defers lance_dataset_close until all active FFI calls exit, preventing concurrent use-after-free.

Upstream Coordination Tracker

  • lance-format/lance-c#163 — Strip Linux/macOS release shared libraries and restrict exported symbols to lance_*; accepted upstream in #201. Windows dependency exports are separately tracked in #214. Actual package sizes remain artifact-specific.
  • lance-format/lance-c#164 — Remove unused libssl-dev from Linux release image and verify liblance_c.so DT_NEEDED allowlist
  • lance-format/lance-c#165 — Add lance_dataset_clone_handle (plus optional lance_clear_last_error / lance_c_version) for immutable *Dataset snapshots and lock-free concurrent reads
  • lance-format/lance-c#51 — Full LanceNamespace C ABI coverage and namespace-backed Dataset operations
  • lance-format/lance-c#103 — Official lance-c prebuilt binary platform matrix and Linux glibc baseline

Upstream clone-handle and namespace C ABI delivery is complete. Go clone adoption is accepted in #2 across delivered mutation, schema, index, and compaction paths; namespace adoption is accepted in #3. Custom namespace adapters use the shared dispatcher delivered by lance-c#197, which closes #180; Go implementation and paired artifact adoption are accepted through #38; the real cloud refresh boundary is accepted in #3 through #37.

Implementation Roadmap

Merged delivery: foundation #4, core Arrow/read/write APIs #7, loader/cache/artifact generation #8, Windows loader/callback/PE plumbing #11, immutable row mutations #10, and verified native CI installation caching #14. Mutation acceptance covers unchanged receiver snapshots, independent close, concurrent reads, and close waiting, with native/race tests on NieR and hosted CI. The cache passed cold build/save and exact-key warm restore with ABI/native/race gates retained.

Checked items reflect merged work; combined unchecked items may be partially implemented. Default Linux amd64 bundle #12 is merged at 3ac8d31; independent public-proxy default/external consumers, native integration and race/checkptr acceptance passed on Ubuntu 24.04/glibc 2.39. External consumers still fetched the artifact archive during go get. Version/tag/branch APIs #15, typed vector queries #16, index lifecycle/FTS/MinHash #17, and schema/metadata/config/BatchUDF #18 are merged with scoped NieR runtime evidence and hosted integration checks. CI repair #19 passed cold-build and exact-cache warm native/race runs and is merged.

Compaction planning/execution/commit and cleanup #24 are merged at 390dac69d7e73725dcb239ddbaf96b8d22534d74. NieR accepted the final pointer-array repair, 233 bindings, full native/race/checkptr suites, and fresh-bundle execution without skips. Main CI 43062789567 and Windows 43062789599 succeeded. OptimizeIndices and multi-base registration are now delivered below, completing this maintenance item.

Transactions/fragments #21 are merged at 39ce155150ea824cdb6c7377418960337ecb8c21, including the paired native 07cd7d0 bundle and compaction integration. NieR accepted 318 bindings, full native and race/checkptr suites, and fresh-bundle execution without skips; the earlier independent public default/external consumer used the same native bundle. Session option-pointer repair #25 is merged at 357aa7497c83bc9862b01bd0128b21c0f65f874e, with its public Moka regression and full NieR acceptance; main CI and Windows checks passed. Post-merge main CI and Windows checks for #21 succeeded. Blob #22 is merged at 4e0902fbe1cef9fdf40fcc1145e7de7daa6d43db: all 11 hosted checks passed, including 346-symbol ABI verification, native/race suites, retained-stream ownership, delayed payload I/O errors, and the public cold-cache path. Its post-merge main CI and Windows checks succeeded. Raw file #23 is merged at 0842d4a5b3c6662088b0cedf0a10de62e5fe57c0: all 11 hosted checks passed, including 412-symbol ABI verification and native/race execution of all nine file integration tests with no skips. Its post-merge main CI and Windows checks succeeded. Distributed indexing #26 is merged at 6a3880c86f1a394053df9d5981ddb9be18f58081: all 11 hosted checks passed, including 449-symbol ABI verification and native/race execution of training/build/metadata transport/commit/reopen/progress tests without skips. Its post-merge main CI 43073794900 and Windows checks succeeded. OptimizeIndices #28 is merged at 99c16af8fd28a7e6345763848010c5ad01930c41: all 11 hosted checks passed, with 505 bindings and native/race execution of transported plan/task/result merge and commit, reopen/query oracles, immutable snapshots, option semantics, and no-op/foreign-plan behavior without skips. Post-merge main CI 43074667864 and Windows 43074667807 succeeded. Raw-file default stream parameters and upstream validation are tracked in #27. Multi-base registration #29, shared file Session caches and packed batch Blob writes #30, and scanner controls #31 are merged in main 589da8bd631e89eff652fafcd952ad84f88551c2 (522 bindings). Each exact PR head passed independent source review, all 11 hosted checks, and NieR tidy/vet/ABI/native/race+checkptr/fresh-bundle gates with zero skips. Their combined main 589da8b acceptance also passed: 522 bindings, 218 top-level/140 subtests in CGO0 native mode, 210/127 with external-tag race/checkptr, and 129/70 on fresh bundled root/file execution; zero failures or skips. Scope remains Linux amd64, Go 1.23.12, and the pinned 07cd7d0 library. Native namespace adoption is merged through PR #32, and the dynamic storage provider/accessor bridge is merged through PR #36. Custom Go namespace dispatch and the owned native WriteParams entry points from upstream #197/#198 are integrated and paired with the public native88 bundle in merged PR #38, including retained CommitLock. Exact9912 and its identical main8a75 tree have complete Linux native/race/public-consumer acceptance. Subsequent cloud refresh, Windows native and six-platform public-distribution acceptance are recorded below with their exact source and runtime scope.

Phase 0: Repository Scaffolding, Loader & Artifact Pipeline
  • Initialize root go.mod, artifacts/<os>-<arch> companion modules, and package skeleton
  • Implement internal/loader (LANCE_LIB_PATH, -tags lance_external_lib, content-addressed zstd extraction with flock, dlopen symbol verification)
  • Add scripts/sync-lance-c.sh and scripts/check-abi-symbols.go against lance.h
  • Configure GitHub Actions CI (CGO_ENABLED=0, -race, multi-OS/arch matrix) — merged #40/#44 cover six native targets; all supported race targets pass. Go does not support Windows ARM64 race, which is explicitly recorded. This accepts CI configuration and supplied-library testing, not cold public distribution.
Phase 1: FFI Core, Pure-Go Arrow cdata & Core Read/Write MVP
  • Implement internal/handle sharded handle table and internal/ffi binding layer (thread-locked error extraction, static callback trampolines, error code mapping)
  • Implement internal/cdata zero-copy import/export (ArrowSchema, ArrowArray, ArrowArrayStream, runtime.Pinner integration) covering primitive, temporal, decimal, binary/utf8, list/fixed-size-list, struct, map, dictionary, sliced offsets, and nested null bitmaps
  • Implement Session, Open, Write (Create / Append / Overwrite + full WriteParams), and dataset metadata inspection (Schema, Version, LatestVersion, CountRows, Validate, DataStatistics)
  • Implement Scanner (Scan, Records(ctx) Go 1.23+ iter.Seq2, ToArrowReader, projections, SQL & Substrait filters/aggregates, OrderBy, Limit/Offset, WithRowID/WithRowAddress, scan stats) and Dataset.Take / TakeRows
Phase 2: Immutable Mutations, Schema Evolution, Versioning & Distributed Writes
  • Integrate lance_dataset_clone_handle for immutable-snapshot return semantics across all mutating APIs
  • Row-level mutations: Delete, Update, MergeInsert (upsert), Merge, Truncate
  • Schema & metadata evolution: AddColumnsSQL, AddColumnsNulls, AddColumnsStream, AddColumnsBatchUDF (with checkpoints), AlterColumns, DropColumns, schema/field/table metadata & config updates
  • Time travel & branching: Versions, CheckoutVersion / CheckoutTag / CheckoutBranch / CheckoutRef, Restore, Tags CRUD, Branches CRUD
  • Distributed write primitives: Fragments inspection, uncommitted WriteFragments, Transaction inspection/Protobuf serde, and CommitTransaction
Phase 3: Vector Search, Full-Text Search, Scalar Indices & Distributed Indexing
  • Vector kNN search (Nearest: L2 / Cosine / Dot / Hamming, nprobes, ef, refine_factor, pre/post-filtering, row address allowlist/blocklist)
  • Full-text search (FtsQuery: Match / Phrase / Boost / MultiMatch / Boolean) and MinHash LSH search
  • index package: Create, list, stats, prewarm, and drop vector indices (IVF_FLAT, IVF_SQ, IVF_PQ, IVF_RQ, IVF_HNSW_*) and scalar indices (BTree, Bitmap, LabelList, Inverted, NGram, ZoneMap, BloomFilter, MinHash, RTree) with progress callbacks
  • Distributed index building: IVF/PQ/RQ training, shard SegmentBuilder, and CommitIndexSegments
Phase 4: Multimodal Blob v2, Raw File v2, Table Maintenance & Dynamic Cloud Credentials
  • BlobFile (TakeBlobs by positions/addresses/IDs, descriptor builder, planned range reads) implementing io.Reader, io.Seeker, io.ReaderAt, and io.Closer
  • file package: Raw .lance v2 Reader and Writer
  • compaction & maintenance: single-node CompactFiles, distributed Compaction V2 (Plan / ExecuteTask / Commit), OptimizeIndices, Cleanup / CleanupOldVersions / CleanupExplain, multi-base-path registration
  • Cloud object-store options passthrough and StorageOptionsProvider callback bridge for dynamic IAM/STS credential refresh
Phase 5: lance-namespace Integration & Production Readiness
  • namespace package: Namespace interface, OpenAPI request/response structs, DirectoryNamespace, RestNamespace, RestAdapter, Go callback namespace adapter, and DynamicContextProvider
  • Namespace-backed dataset operations (OpenFromNamespace, WriteToNamespace, managed commits, table-scoped credential refresh)
  • Cross-language compatibility test suite (lance Rust / pylance / lance-java / lance-go), long-running native RSS leak benchmarks, documentation, and initial preview release (accepted scope: storage2.1 on Linux amd64; 30-minute RSS measurement, not leak absence; six-target preview tag acceptance)

Remaining delivery actions

Outcome Next action and concrete dependency
Integrate accepted six-platform distribution Complete: PR47 merged as 0e0d0ef4f898f5baf461e6a66ec3eea96629e491, with the accepted 807cb84 tree unchanged. Windows distribution issue #9 is closed.
Integrate preview documentation PR48 targets main at 956bee25817c4973190a946a3898bfea75910d5b. Independent review findings are fixed, PR47 integration conflicts are resolved, and obsolete approval requirements are removed. Await its CI results before the authorized exact-head merge.
Publish and accept the preview version Bind the final merged source and authorized version, publish, then run both public-consumer workflows against that tag with fresh caches. Require all six targets / 18 modes and cleanup, and retain the checksummed release evidence. Existing pseudo-version results cannot prove tag resolution.
Complete RTree and raw2.2 parity Official v15.0.0-beta.20 contains the RTree reload and raw Blob-reader fixes. The existing C owner is authorized to update the accepted core15/Rust1.95 candidate and proceed through normal C PR/CI/merge acceptance. Whole PR149 remains held; RTree C/Go API delivery and raw2.2 public-reader acceptance are still pending. Current Go native modules remain unchanged; no fork/backport, lance-c package publication or Go module changes are authorized. See #6 for exact identities and acceptance boundaries.
Finalize the official upstream artifact matrix and baseline policy The glibc2.28 baseline is adopted for future generic Linux x86_64/AArch64 official packages, preserving Debian12 variants. Follow lance-c#103 and its existing owners for acceptance of author-maintained PR50 integration and the actual supported installation/package paths; the policy decision is complete. Six Go targets already have public runtime acceptance on their documented hosts. Older macOS/Windows and musl coverage are unclaimed limits, not newly introduced preview prerequisites; a future lower-floor adoption needs validation of the actual Go payloads.

Current-ac263 storage2.1 compatibility and the long-run RSS measurement are delivered within the scopes below. Cloud refresh has accepted historical-native88 evidence. A current 807cb84 / ac263 runner has additionally been built and audited on NieR, but live replay has not run: the existing scoped AssumeRole and GetFederationToken issuers both return AccessDenied. No IAM mutation or broader credential transfer was made. This specific credential dependency does not block documentation, PR integration or the already-prepared release evidence.

Accepted evidence and remaining acceptance

Main is now 0e0d0ef4f898f5baf461e6a66ec3eea96629e491, including the accepted ac263 SDK and six-platform public distribution from PR47. Its tree is identical to accepted head807cb84. Historical evidence below retains its exact source scope. Preview release and post-tag acceptance remain incomplete.

  • Native88 integration accepted on NieR at 9912e80ab7ded556160adf211c66bb375528a7c2. Source 88c07cf8ccafebf579e769416251be370b45180e, header c23406d5c683133eb647cfeb150fa05182ffc8ba7da283b3902dbb4bf0926c6c, stripped library 20b2b63598fbca3f83146061f8ef4d2f616effb359af9ae4a01fd9afe9dad1d8 / 322169936 bytes. The native artifact has 876 exports; Go binds603, with14 production FFI callbacks. Six gates passed: tidy/vet/ABI, CGO0 full344/255, external race/checkptr336/242, fresh bundle205/157, zero skips. Required tests69/69 (bundle50/50). Independent public root and namespace-only consumers verified exact archive source, compressed payload, default embedding/extraction and external non-embedding/zero extraction. This accepts retained CommitLock, native write options, custom namespace, ordinary index progress and staged fragment publication at that object. All11 hosted checks passed and PR #38 is merged; #20/#27/#35 are closed with their full scoped acceptance recorded. Evidence: /home/xuanwo/Code/lance-go-acceptance/artifacts/native88-go-9912e80/, verified13,404-entry manifest.
  • Native C/C++ fragment mutation consumers accepted. Fixed task-only producer 0cb8a35c9a69d12c33d104c0e64f4672f541e33b generated ten independent fixtures. After producer exit, an independent manifest audit verified100/100 literal profile tags, version5/10rows/3fragments, preserved field IDs0/1/3/4/5 and stable row ID2 for id3. Both unchanged full native88 C11/C++17 mains ran once against separate five-fixture sets and passed all add/merge/update/replace/delete publication paths. Runtime loader traces resolve to the retained88 library. Evidence: native88-fragment-consumers-0cb8a35/. The predecessor borrowed-string manifest defect and its failed evidence remain preserved; the clean rerun closes that gap.
  • Cross-language storage2.1 accepted;2.2 unverified. PR #34 merged at 11b1dd91337756348ce999c3cd652022ef0b2e59; 3e36776 passed56/56 with native07. Exact9912 rebuilt Go with native88 and reused verified foreign SDK binaries/input blobs: another56/56,4/4 producers,16/16 reads,6/6 bidirectional append paths and literal goldens. Evidence in compat-3e36776/ and native88-go-9912e80/. The30-minute RSS run at1150683 and short integration at0425a7f remain bounded workload evidence, not proof of no native leaks.
  • Current ac263 long-run lifecycle/RSS measurement accepted within its measurement scope. On NieR, SDK 0d151adcbcaf5d7e13665f8f8a81f062d81687ff with retained raw ac263 completed two minutes of warmup plus 30 measured minutes: 181 samples over 1,800,009 ms, all eight workloads each 189,908 completed operations, zero errors, and sampled callback handles always zero. Parent independently verified the67-file manifest and recomputed the primary and sensitivity journals. RSS rose from369,152,000 to503,996,416 bytes; late-minus-early window mean was111,818,343 bytes. Deliberate256MiB native Take retention produced a measurable RSS response, but releasing it did not immediately reduce residency. Growth attribution remains unknown: these results neither prove a native leak nor establish leak absence or a production memory budget. Evidence: /home/xuanwo/Code/lance-go-acceptance/artifacts/rss-ac263-0d151/; parent audit /home/xuanwo/Code/evidence/lance-go-issue-1/rss-ac263-parent/ACCEPTANCE.md. The current-input long-run measurement is delivered; the combined Phase5 delivery was subsequently completed by the preview tag acceptance below.
  • Real S3 rotation accepted in both modes; #37 merged. Exact runner 59a427e5e1f1ffdb3404f15e185d0ca67f140c2e passed against real AWS S3 on2026-11-11 UTC using retained native88. Public owners closed before refresh; retained Dataset Delete fetched distinct temporary STS credentials B, committed v2[10,30], preserved v1[10,20,30] and verified both through reopen. Direct mode observed one native fetch; namespace mode observed one vend_credentials=true request through real loopback HTTP to an operator-owned descriptor, followed by a non-vend reopen using B. Independent broker/report/timing audits passed; both unique prefixes were cleaned6objects→0. #37 merged at 2e99b85c417cb281ef75e103f8fba0029ae110e8 with11/11 checks; #3/#5 are closed. Evidence: cloud59a-live-direct-accessor-20261111T233854Z-a57709/ and cloud59a-live-namespace-20261111T234035Z-1de851/. This is self-hosted namespace vending, not commercial service acceptance, namespace-managed publication or expired-A rejection.
  • Four Unix public companion targets accepted at 4e377bcf67bf107b2986261c2454ab2cdc2eb306; PR45 merged as a79eaf1a4a04a929b9b3e30e1bcc3d830814c1eb. PR #40's reviewed native/strip/candidate pipeline passed all four targets and merged at 27a8c1eea1ca56b34867f6f5372266973d793fbf. Linux arm64 and macOS amd64/arm64 data modules were published and independently verified through public proxy/sumdb at 43fcbbee5161f1e461cbd2804ef47f243b872795; Linux amd64 retains the accepted ac5955a095d0 module. Run43195628310 repeated the earlier b813 acceptance and completed 4/4 jobs and 12/12 default, namespace-only and external consumer executions, including final cleanup, with Go1.23.0 and CGO_ENABLED=0 on native Linux amd64/arm64 and macOS amd64/arm64 hosts. Each root archive matched all291 Git files. Independent target caches verified exactly one selected artifact archive for default/namespace builds, precise binary embedding and extracted library hashes, namespace-only native use without a root-package dependency, and external execution with no artifact linkage/embedding/extraction. Ordinary setup go-get downloaded all four pinned archives; external target builds happened to download zero, which remains an observation rather than a zero-download contract. Tested runtimes are Linux glibc2.39 and macOS15.x; older glibc/macOS and musl are not accepted. Parent verified all four full evidence ZIPs against GitHub sizes/digests and independently reconciled metadata, download inventories, cache reports and runtime stdout; retained current-head evidence is /home/xuanwo/Code/evidence/lance-go-issue-1/unix-public-4e377bc-hosted/ (earlier b813 evidence remains preserved). Earlier driver failures and their evidence remain retained: setup/target cache separation (99cf6d1), root mod/dep identity parsing (6681b13), and read-only Go-cache cleanup (b8132c7) close those observed defects. Independent source reviews found no blockers for adoption, cache boundaries and identity parsing. PR45 is merged; no local push or dynamic-CRT run remains pending for it. The superseded dynamic-CRT and Toolhelp failures retain their original logs and audits under windows-unix-adoption-d9835b1/ and windows-ea2c910/. Accepted static-CRT candidates and six-platform public consumers are recorded below; those later results supersede the old Windows distribution gate.
  • Windows static-CRT native and unpublished candidates accepted. PR #46 source c8373c9911a8fa3c9157799261c9cc596bfd278b passed all13 checks in run43197858510. Both architectures pass the C lifecycle/loaded-module guards,603-binding ABI,878 matching public exports,full external353+270 and supplied-default361+283 tests with zero failures/skips; amd64 race/checkptr353+270 passes,while ARM64 race is unsupported. Independent PE audits verify16 OS-only direct imports and empty delay-import directories. Both candidate verifiers and supplied-library roundtrips pass; independently decompressed payloads equal the staged DLLs byte-for-byte. Full ZIPs match API head/size/digest and are retained with118-file manifest,ACCEPTANCE.md and AUDIT.json under /home/xuanwo/Code/evidence/lance-go-issue-1/windows-c8373c9/. These static-CRT results resolve the prior VCRUNTIME140 eligibility blocker for these exact candidates. They do not publish modules or establish cold public consumers. PR46 integration checks passed and it merged as fcd76d9. The subsequent public-consumer acceptance and completed PR47 integration are recorded below.
  • Bulk offset adoption is delivered in #43, resolving #39. The previous native88 per-element traversal could be quadratic. Upstream#209 is closed by #211; fixed source ac2634c553b450c7787577ffe964fc2b283b8334 provides both bulk-copy entry points and preserves the pinned core/dependency lock. PR43 at626af451 pairs the two bulk-copy bindings with public companion ac5955a. Nativeac263 has878 matching declarations/exports and both full C11/C++17 fragment mutation consumers passed, including bulk-copy oracles. Kimi found no source blocker; explicit non-nil empty-result checks were added. Exact candidate626af451 passed NieR tidy/vet/603-binding ABI, full CGO0 native358 top/268 subtests, explicit race+checkptr350/255, and fresh bundled root/file/namespace207/157, with zero failures or skips. Both bulk tests and fragment publication/reopen ran in all applicable lanes; the parent independently recounted raw logs. Fresh public root consumers matched274/274 Git source blobs and verified exact ac263 payload embedding/extraction, external non-embedding/zero extraction, and standalone namespace loading. External setup still downloads the artifact module. Evidence: bulk-ac263-go-626af45/ (222-file verified manifest), with parent raw-log readback and independent audit. All 11 hosted checks passed, including native and race. PR43 merged at 06fda6ca8529150830e62aa0bae6c0519fc52ba8 on 2026-11-12; the merged tree matches accepted626. Both paths now perform a cardinality query plus one native bulk copy. No bespoke Go Roaring decoder or measured speedup claim is introduced.
  • Preview delivery is complete within the accepted scope below. Release procedure PR #41 and documentation PR #48 are merged. The published preview tag passed the actual six-target/18-mode public-consumer workflows. RTree and raw v2.2 Blob-reader limitations still require downstream adoption and acceptance (#153/#167); official beta20 now contains the upstream fixes.

Retain separate evidence for implementation, source review, exact-object runtime acceptance, merge, public distribution and release readiness. No whole-roadmap completion is claimed.

Public distribution accepted (2026-11-13 UTC): PR #47 at 807cb84305302dc3ff3b40809d817a876efd2909 passed all 23 checks. Public root v0.0.0-20261112185256-807cb8430530 passed default, namespace-only and external execution on all six targets: four Unix targets and two Windows targets. Independent audit verifies all 18 runtime outputs, fresh target caches, 96 verifier reports, source/archive identity, selected payloads and loaded Windows DLL identities/CRT guards. Setup downloads and cold target downloads are measured separately; external excludes artifact linkage/extraction but makes no zero-download claim. Evidence: /home/xuanwo/Code/evidence/lance-go-issue-1/public-807cb84/ (266-file manifest).

Both Windows modules are published at v0.0.0-20261112113154-9496baf852b1; the SDK keeps metadata snapshots while complete payloads remain at the retained publication commits. PR46 is merged as fcd76d935009abb3d1123c82281034ba310c134b. PR47 merged as 0e0d0ef4f898f5baf461e6a66ec3eea96629e491 after all 23 checks passed and GitHub reported MERGEABLE/CLEAN. Its tree is identical to accepted head 807cb84. This accepts the tested hosts (Linux glibc 2.39, macOS 15.x, Windows Server 2022 amd64 and Windows 11 ARM64), not older platform floors, a release tag or whole-roadmap completion.

Current-ac263 compatibility accepted (2026-11-13 UTC): SDK 807cb84305302dc3ff3b40809d817a876efd2909 with the published ac5955a library (native ac263, SHA-256 e5571d6da16e81a0434f2509ade5262785b63df0b37ff7155537642841dd9378) passed all 56 storage2.1 cells on Linux amd64: four producers, 16 cross-readers and six bidirectional append paths. Parent independently matched every cell stdout to committed literal goldens and every copied log to the verified remote manifest (3110 entries); zero missing cells, nonzero exits or mismatches. Foreign SDK pins are unchanged; Python was rebuilt in isolation to the original 13-package freeze after detecting drift in the retained venv. Evidence: NieR compat-ac263-807cb84/; parent compat-ac263-807cb84-parent/. This does not cover storage2.2, other platforms, cloud refresh, or a release tag. Preview documentation is merged in PR48; actual tag acceptance is recorded below.

Preview tag acceptance completed (2026-11-13 UTC): v0.1.0-preview.1 is published from PR48 merge 0e186d62d0fe3758e190502f9273df7dd7409c55 (tree 8816f47b6112dbf9812a978cbae2e0f56862ecb7). The exact tag passed six native targets and all 18 default/namespace-only/external modes in Unix run43404796150 and Windows run43404798900, both attempt1. Independent acceptance verified terminal exits, tag/source/root-module identity, fresh target caches, cleanup, six API ZIP sizes/digests and all262 extracted files against their ZIP members. Durable post-tag evidence is attached to the release (lance-go-v0.1.0-preview.1-public-evidence.tar.gz, SHA-256 11b16611b17f5cbb5911720934e4e7581c4b2f2f9591bf2fea90cd0e01519209). Storage2.1 compatibility remains 56/56 Linux amd64 cells; the 30-minute RSS run is an accepted measurement with observed growth, not proof of leak absence. RTree API, raw-v2.2 downstream reader acceptance and official upstream platform integration remain open; beta20 fixes and adopted glibc2.28 policy do not themselves complete those deliveries. This does not close the whole roadmap.

Lingua principale
Go
Stelle
1
Fork
1
Merge medio
2h 10m
PR unite (30g)
41

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance-go

Tutte le issue di lance-format/lance-go

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.