Tracking: high-throughput Rust ingestion with a three-replica WAL

Aperta
#378 119 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust

Direzione di ricerca

Start by reading the incremental delivery checklist and the listed references on Flink recovery and Fluss replication; this tracking issue names no implementation files or tests. The completed steps cover groundwork, while the remaining work spans replicated WAL, recovery, Lance publication, reclamation and production integration. Done means the listed validation passes across real processes and durable storage, with sustained ingestion and recovery behavior measured.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

Object-store WAL writes can put remote request latency and small-object overhead on the ingestion ACK path. A memory-only WAL removes that cost but cannot preserve acknowledged writes across process or node loss. We want an optional, high-throughput durable ingestion path implemented in Rust within lance-context, with minimal changes to Lance OSS.

This issue tracks incremental implementation and validation. Explicit Rust service paths now exist for durable append and mutable turn ingestion. They remain opt-in; workload-specific production readiness and large-scale qualification are still required.

Architecture and current visibility contract

The mutable-turn service path now separates immutable bodies from current narrow
annotations. The diagram shows that path for one shard; the lower-level immutable
append API has a different ACK/visibility contract, described below.

flowchart LR
    I["Ingress router: forward Arrow bytes"] --> O["Shard owner: ordered apply and point reads"]
    O -->|"Submit original request identity and payload"| L
    subgraph G["One Raft group: three durable replicas"]
        L["Leader + persistent WAL"] --> F1["Follower + persistent WAL"]
        L --> F2["Follower + persistent WAL"]
    end
    L -->|"Quorum-committed records"| O
    O --> S["Bounded active / frozen staging"]
    S -->|"Frozen batch"| P["Fenced Lance publisher"]
    P --> B["Blob: body and narrow-annotation snapshots"]
    B -->|"Compatible selected base"| O
    O -->|"Success after durable quorum and applied visibility"| I
  1. Explicit shard placement. The current mutable-turn path hashes the turn ID
    within its tenant/table scope to a configured bucket and owner. Forwarding
    ingresses share the validated route map and do not keep independent staging
    views. Each physical annotation shard has its own body/narrow table pair and
    publication/recovery progress. The immutable append contract retains its
    session-key placement. Automatic session placement, shard migration and
    rebalancing are not supplied by these explicit shard APIs.
  2. Durable and visible mutation ACKs. Three replicas persist the log on
    independent durable storage. A successful turn mutation requires majority
    persistence and ordered application to the owner's queryable state. A raw WAL
    append receipt alone is not that success. Raft supplies election, ordering and
    committed-prefix safety; it does not publish Lance manifests. The storage
    fault model tolerates one replica's loss, not loss of a durable quorum.
  3. Field updates and immediate point reads. Patches preserve unspecified
    fields; status/score have explicit keep/set/clear operations, and labels have
    set addition/removal. Applications merge against the ordered current value.
    Point reads resolve active, frozen, then the compatible Lance base, projecting
    narrow columns only and never scanning the WAL suffix. Owner fencing and
    recovery barriers protect cross-ingress reads. Materialized status filters
    currently have a different freshness contract: they do not include active or
    frozen updates. CAS and cross-shard transactions are not implemented here.
  4. Batch publication without losing later updates. Freeze one bounded
    generation at a source cut while later updates enter a new active generation.
    Existing annotation rows use native narrow-column rewrites; newly ingested
    bodies are appended separately and bound to the selected annotation snapshot.
    Confirm the exact fenced publication before installing its base and retiring
    only its frozen generation. Never delete active records by turn ID when an
    older batch finishes. Compaction is separate bounded maintenance, not a
    per-update rewrite of the immutable body.
  5. Recovery, exact retry and reclamation. Recovery uses the selected Lance
    state, persisted request outcomes and the required WAL suffix. Retained exact
    producer/sequence retries return their original result; an unknown outcome
    requires the same identity and bytes, not a newly identified mutation. A
    producer has at most one unresolved request, and the bounded latest-result
    contract does not retain arbitrary historical retries. Unknown publication
    results are reconciled against the original publication. Source GC also
    respects readers, recovery and replica/snapshot dependencies; a materialized
    cut alone does not permit deleting every preceding log record.
  6. Bounded resources and measured copying. Staging, caches, payloads, native
    workspaces, disk admission and replication queues have explicit budgets;
    pressure must leave publication and reclamation able to make progress.
    Replicas keep Arrow payloads opaque and ingress forwarding shares byte
    buffers. The current owner still validates by decoding on preparation and
    decodes again during ordered apply; this is not end-to-end zero-copy.
    Framing, integrity checks, native storage and kernel I/O also need measurement.

The existing immutable append API remains separate: its durable WAL ACK can
precede Lance visibility. Its readers and publication checkpoints follow that
binding's contract. Neither this issue nor an annotation-owner configuration
silently changes an existing immutable binding's semantics.

The implementation uses Rust raft-rs and raft-engine with Lance's existing
publication APIs. Useful references remain Flink's checkpoint/replay and bounded
flow control, and Apache Fluss's bucket ownership, batched replication and
asynchronous lake publication. Fluss uses leader/follower replication with ISR;
this design does not imply that Fluss uses Raft. Independent shard publication
does not create a global WAL position or a cross-shard atomic snapshot.

Incremental delivery

Use small PRs with Refs #378 in their descriptions. Keep this tracking issue open across individual implementation PRs; merge each only after its checks pass.

  • Lance 13 dependency/API upgrade: #374.

  • Rust Arrow IPC allocation/buffer-sharing and durable disk experiment tools: #375.

  • Stable session placement and scoped receipts: #376.

  • Shared node-wide proposal admission: #377.

  • Configurable maintenance headroom and pressure regressions: #379.

  • Validated Arrow IPC decoding with shared payload ownership: #380.

  • Shared opaque record envelopes and bounded payload validation: #381.

  • Move owned HTTP rollout payloads into core records without a conversion copy: #382.

  • Honor registered rollout dataset locations on cold open, metadata and deletion: #383.

  • Immutable ingestion contracts shared by ingress and materializers: #384.

  • Prevent concurrent dataset creation from overwriting committed data: #385.

  • Shared Arrow schema fingerprints for ingress and materializers: #386.

  • Guarded Rust experiment process runner and optional-feature CI: #388.

  • Read-only Rust process and cgroup resource observation: #389.

  • Optional process-owned TCP socket observation, with descriptor ownership, namespace and connection-state tests: #438.

  • Bounded vectored frame writes over shared payloads: #393.

  • Measure WAL component disk footprint from actual files: #397.

  • Isolate physical log read caches by engine lifetime, including rewrite reads: #398.

  • Shared durable Raft log storage, sync failure fencing and recovery contracts: #399.

  • Replicated append, durable quorum ACK, retry deduplication and leader failover: #400.

  • Local durable client journal, shared group commit and recovery tests: #401.

  • Reclaim obsolete client journal rewrite bytes before physical admission fences, with synchronized replacement, failure fencing and SIGKILL recovery: #430.

  • Durable local Raft checkpoint storage boundary: #402 (application snapshots and remote transfer remain separate work).

  • Atomic received Raft snapshot storage: #403.

  • Application checkpoint files, bounded transfer and retained-replica recovery: #404 (including interrupted transfer and full process restart tests).

  • Durable canonical table bindings and publisher ownership epochs: #405 (external Lance commit fencing remains separate work).

  • Bounded quorum queries for publisher bindings, accepted pages and producer tails: #406.

  • Immutable publication plans, fixed transaction targets and independent source-group selections retained across takeover and snapshots: #407 (source pins and external commits remain separate work).

  • Catalog quorum attestations and durable source-quorum pins across takeover, snapshot transfer and full process restart: #409 (source release and external publication remain separate work).

  • Immutable prepared Lance fragment descriptors, exact retries and real-file recovery across publisher takeover and full process restart: #410 (manifest authority and source retirement remain separate work).

  • Select immutable staged Lance manifests through the catalog quorum, with exact head reservations and recovery after lost replies, owner takeover and full process restart: #412 (outcome advancement and source retirement remain separate work).

  • Persist durable publication outcomes from the exact catalog-selected manifest, recovering identical confirmations across takeover and full process restart: #414 (source closure and retirement remain separate work).

  • Close source pins with durable publication fences, reject delayed repins and preserve exact closure retries across snapshots and restart: #415 (payload retirement and physical reclamation remain separate work).

  • Retire closed source payloads from new checkpoints while retaining original retry receipts and retirement boundaries across restart: #416 (physical reclamation and catalog advancement remain separate work).

  • Use optimized SHA-256 for opaque WAL payload validation while preserving persisted digests and record compatibility: #417.

  • Opt-in Lance object-store I/O metrics through the existing recorder, with core builds supported when metrics are disabled: #421.

  • Source-group closure attestations after a quorum read, with separate signing keys and checkpoint/restart recovery: #423 (catalog advancement and physical reclamation remain separate work).

  • Advance catalog source cursors only after every selected source attests durable closure, preserving exact retries and managed Lance recovery: #424 (physical reclamation remains separate work).

  • Bounded collection of obsolete application snapshot files, preserving installed recovery images and captured readers across shared groups: #426 (native log reclamation and maintenance scheduling remain separate work).

  • Explicit physical log segment reclamation with node-wide sync barriers and failure fencing: #427 (maintenance scheduling and replica replacement remain separate work).

  • Bounded managed-manifest history retirement with protected publication dependencies and explicit retired-retry boundaries: #428 (metadata only; object deletion and production reader integration remain separate work).

  • Online replica replacement, durable-volume-loss recovery and operational recovery integration.

  • Lance materialization, fenced publication and durable source checkpoints: #450, building on the catalog/source protocols above. Sustained capacity remains under qualification.

  • Real Lance transaction-proof cleanup, protected version floors and delayed fixed-target commit regressions: #408 (coordinated GC remains separate work).

  • Safe logical retirement and physical reclamation under concurrent activity.

  • Bounded durable replica workers with shared node credits, cancellation ownership, responsive cached status and joined shutdown: #441 (network transport, snapshot transfer and maintenance integration remain separate work).

  • Producer batch sizing and end-to-end throughput/latency qualification with unchanged correctness gates.

  • Bounded reusable producer slots with durable generation fencing, exact-tail close, and reuse only after signed source retirement: #434 (optional for fresh groups; administrative authorization, SDK integration and sustained churn qualification remain separate work).

  • Replica-pressure coordination, producer-state lifecycle and bounded steady-state operation.

  • Reusable Lance manifest reader/writer adapter over catalog quorum reads and fixed-target durable reservations: #439 (production transport and writer migration remain separate work).

  • Independent managed Lance readers with scoped catalog resolution and rejected write attempts: #440

  • Production integration: visibility API, ownership/catalog integration, authentication and migration/rollback documentation.

  • Release engine directory locks explicitly at final owner shutdown, with duplicated-descriptor and pre-exec inheritance regressions: #465.

  • Distribute one pinned table publication across bounded authenticated Rust preparer services, keeping complete aggregation and manifest authority with one fenced publisher: #466. Independent process recovery has passed; Kubernetes scaling and sustained throughput remain under qualification.

Validation before enabling the path

  • Verify acknowledged records after leader crashes, follower lag, snapshot transfer interruption and full process restarts using real Rust processes and durable storage.
  • Exercise ambiguous replies, stale publishers, concurrent manifest commits and retries; verify complete Lance payloads and source positions after recovery.
  • Run sustained ingestion through publication and repeated reclamation, including controlled publication pauses. Report ACK latency, published throughput, backlog, RSS, allocated disk space and physical write amplification separately.
  • Compare batching, serialization/deserialization, shared-buffer ownership, bounded parallel reads and replication-group placement using reproducible workloads. Short component benchmarks are not evidence of sustained end-to-end throughput.
  • Publish only synthetic, portable workloads and sanitized results. Do not include private deployment identifiers, storage locations, identity configuration, business workloads or raw operational artifacts.

Related: #333 tracks maintenance scheduling in the existing system; this issue focuses on the optional durable ingestion path and its correctness/performance contract.

References

  • Flink state backends and incremental checkpoints.

  • Flink network stack and credit-based flow control (historical design explanation).

  • Fluss LogTablet replication and log lifecycle and Lance lake connector documentation, pinned to the researched revision.

  • Authenticated bounded TLS peer transport shared across Raft groups, with stalled-follower isolation and real three-process crash/recovery qualification: #442 (snapshot transfer, disk admission and service integration remain follow-up work).

  • Durable runtime checkpoint/snapshot installation and physical-maintenance transitions, with captured image ownership and durable completion tests: #443 (network image transfer and node-wide disk admission remain follow-up work).

  • Reserved snapshot completion lane with cancellation and saturated-control regression coverage: #444.

  • Bounded authenticated TLS snapshot transfer with interrupted-receive and full-restart process coverage: #446.

  • Shared node disk admission for logs, producer state, snapshots and reclamation, with restart coverage: #447.

  • Bounded TLS client RPCs with shared physical connection pools and real-process recovery: #448.

  • Rollout ingestion with shared memory admission, session buckets and durable receipts: #449.

  • Recoverable Lance file preparation, fixed-version publication and source retirement: #450.

  • Independent WAL node service, automatic maintenance and six-process restart qualification: #451.

  • Bounded HTTP gateway and journaled Rust producers with exact-retry and Lance recovery qualification: #453.

  • Add bounded materializer workers and independent managed readers (#455).

  • #457 — Idempotent managed rollout bootstrap with persisted genesis identity and crash recovery.

  • Reconnect read-only RPCs after a reused pooled TLS stream disconnects, preserving the original deadline and admission credits: #459. Mutating requests still require explicit original-identity reconciliation.

  • Local runtime admission observations with separate request/byte rejection counters and real-quorum pressure regressions: #460.

  • Expose local runtime request/byte admission observations through node diagnostics without changing readiness: #461.

  • Qualify a single managed table with multiple independent session producers, including full per-producer payload scans and an aggregate table row-count check.

  • Parallel file preparation across workers for one table, with one fenced publisher aggregating a complete bounded set of fragment descriptors: #466. Scaling and sustained capacity remain under qualification.

  • Handle and qualify larger inline payloads, including the 40/120 MiB cases in #222. The current replicated-record limit is 8 MiB; raising a workload size alone does not cover peer framing, bounded memory, journal/snapshot recovery, or materialization.

  • Compose bootstrap, authenticated ingress, materializer worker and managed-reader roles into a Rust application with executable recovery coverage: #462. Dynamic registry discovery and sustained scaling remain separate work.

  • Expose bounded cumulative materializer phase timing and outcome counters, with documented overlapping-duration semantics: #463. Instrumentation only; no throughput improvement claimed.

  • Add opt-in bounded streaming file preparation within one worker, preserving complete descriptor aggregation and cancellation ownership: #464. Default off; distributed same-table preparation and performance qualification remain separate work.

  • Add opt-in replica spreading for pinned WAL source reads, preserving exact source validation and all-voter fallback: #468. Recovery and single-reachable-replica tests pass; controlled throughput comparison is pending, default remains first voter.

  • Overlap bounded replication with leader persistence (default off; no throughput claim): #469

  • Reuse verified uncompressed engine read blocks to remove one allocation and copy on cache misses: #470. Local durability and recovery checks pass; end-to-end performance qualification remains separate.

  • Return already committed WAL fetches before unrelated subsequent persistence, preserving append ACK barriers and complete payload validation: #471. Real-file failure and process recovery checks pass; end-to-end performance qualification remains separate.

  • Preserve bounded source-scan progress across worker claims: #472

  • #473 — Separate local fetch queue and service timings to diagnose ingestion stalls.

  • Validate preparer source pins concurrently within the existing fetch limit, completing every validation before file writers start: #474. End-to-end performance qualification remains separate.

  • Bound CI dependency installation and use the official archive so ingestion recovery tests can execute reliably: #475.

  • Rotate source priority using the completed publication version so larger records cannot be repeatedly skipped; cover publisher takeover and durable-log restart: #476.

  • Allow explicitly budgeted publication source limits up to 1 GiB while preserving the default and immutable policy identity: #480. Sustained performance qualification remains separate.

Query visibility and current follow-ups

The optional session-read API now combines acknowledged WAL rows with published Lance rows. It captures committed WAL metadata, reads retained payloads, then opens a sufficiently recent Lance snapshot and removes overlap by immutable row ID. Readers must consume the complete stream and propagate errors. Different sessions and bounded distinct-record fetches can run concurrently; the WAL-to-Lance visibility handoff must remain ordered. This is not a claim of sustained query capacity or a migration of the existing production API.

  • Query acknowledged session rows before Lance publication: #481.

  • Batch small publications with a bounded flush wait: #482.

  • Expose append execution failure causes: #483.

  • Prefetch distinct session WAL records within shared memory admission: #485.

  • Replay checkpoint compaction after native-log recovery: #486.

  • Admit native cleanup using live rewrite input: #488.

  • Filter session metadata at its WAL group: #489.

  • Bound published session scan batches and storage prefetch: #492.

  • Support compaction publication through the existing catalog protocol: #496.

  • Compact eligible small files through managed publication: #498.

  • Preserve bounded append response diagnostics in the recoverable HTTP client, without changing retry identity or intent advancement: #500.

  • Recover compaction after publisher termination: #504.

  • Configure published session scan concurrency: #508.

  • Schedule bounded compaction while WAL data is awaiting publication: #511.

  • Keep failed maintenance eligible on the next scheduler tick: #512.

  • Preserve follower disk headroom for maintenance: #514.

  • Reuse verified materializer payloads without another digest pass: #516.

  • Preserve row identity in newly created managed ingestion tables: #517.

  • Maintain native session indexes during publication: #518.

  • Identify failed WAL client RPC exchanges: #520.

  • Filter unindexed session fragments before payload reads: #522.

  • Discard disconnected idle streams before dispatching requests: #524.

  • Reuse native log integrity checks for accepted reads: #527. Snapshot pack reads retain payload digest validation.

  • Keep bucket records together within each publication with opt-in byte-balanced preparer assignment: #535. Task/cache identity includes the assignment version; tests cover complete-aggregate publication, retry reuse and WAL reopen. Default striping is unchanged; sustained performance remains under qualification.

  • Qualify combined ingestion, session queries, publication and compaction under sustained load. Configuration changes require throughput, latency, backlog and memory evidence; merged implementation alone does not establish production readiness.

Current turn annotations and mixed ingestion

The current-annotation path adds a stronger visibility contract to the original immutable-ingestion path described above: a successful mutation response waits for both durable quorum commit and ordered application. Point reads through another ingress resolve the owner's current annotation view. Publishing to Lance remains asynchronous.

  • Define label add/remove and status/score set/clear semantics, preserving omitted fields (#567). Apply mutations in committed order and return retained request results for exact retries (#574, #575). CAS remains outside the first implementation.

  • Bound staging to one active generation and at most one frozen generation, with finite reader/cache retention (#569). Point reads address a turn directly and capture a compatible staged value or Lance base; an annotation-only miss projects narrow columns. Ordinary annotation point reads do not scan the WAL suffix.

  • Publish the frozen generation through the fenced catalog protocol, bind its coverage and request results to the selected Lance state, and retire only that generation (#572, #575, #579). Later active updates survive publication and old readers retain their captured views. Recovery reconstructs the remaining suffix; unknown outcomes are resolved using the original publication/request identity (#621, #669, #670, #673).

  • Run authenticated owner/ingress roles with backpressure, bounded file collection and managed small-file compaction (#580, #581, #603, #611). Tests cover concurrent updates, delayed/unknown publication, retained-reader pressure, process recovery and reclamation. These checks do not establish deployment capacity.

  • Add core support for first-create bodies plus current annotations and paired publication/compaction recovery (#656, #658, #659, #666, #672, #675). Re-import of an existing turn preserves its original body and current annotation.

  • Integrate the mixed ordered driver (#676), paired draining/compaction runtime (#677), owner/ingress HTTP roles (#678), and fresh body/annotation bootstrap (#680). Bounded Arrow frame windows retain shared buffers across staging reads and publication. Implementation and recovery regressions are merged; deployment capacity remains a separate qualification.

  • Preserve read-only quorum observation after unknown replies (#679), and keep annotation owners alive while a failed GC listing is resolved (#681). These changes do not resubmit a business mutation after an unknown outcome.

  • Reuse completed compaction checks for an unchanged snapshot (#682) and expose separate annotation observe/fetch/decode/create/patch timing (#683). Stage timings overlap and are not ACK latency percentiles.

  • Bound committed source-metadata pages (#684). Payload admission, fetching, decoding and application remain sequential; merged metadata paging alone does not establish capacity.

  • Use byte targets for managed compaction reads (#652), avoiding batches sized only by row count.

  • Complete sustained performance qualification of the combined mixed ingestion, point-read, compaction and reclamation path. Merged optimizations alone do not establish deployment capacity.

  • Preserve current annotation state and captured readers across native index publication (#685). This core view transition does not by itself enable owner index maintenance or establish index/storage performance.

  • Add opt-in bounded native turn-ID index maintenance through the existing fenced publication and recovery path (#686). Builds and size-tiered merges preserve the materialized cut, request results, body reference and active updates; the option remains disabled by default.

  • Collect obsolete native ID index page files through the existing bounded, fenced collector (#687), protecting raw index UUIDs in retained manifests and held read views. Full repository checks and CI passed; long-term index/storage qualification remains separate.

  • Avoid reserving fetch response bodies for nonempty append requests (#688). Regressions cover concurrent append admission, cancellation credit release and fetch validation. Full repository checks and CI passed; end-to-end throughput qualification remains separate.

  • Include pending image-file reservations in native log reclamation pressure (#689). The real-engine regression keeps the reservation held while reclaiming obsolete payloads and checks admission plus restart. Full repository checks and CI passed; cloud throughput comparison remains separate.

  • Remove the redundant initial ownership lookup from annotation point reads (#690), retaining the quorum fence after resolving the captured view. A real cold-read takeover regression rejects both in-flight and already-stale predecessor responses. Full repository checks and CI passed; deployment throughput comparison remains separate.

  • Add explicit mixed annotation owner/bootstrap APIs for independent physical shards (#691), binding each shard to its durable schema contract and validating every turn before WAL submission. A real RF3 regression covers a stopped owner while another shard drains, retained request results, and WAL reopen. Full repository checks and CI passed; configured ingress routing and multi-worker throughput qualification remain separate.

  • Distinguish annotation WAL append and ordered apply wait observations (#692), exposing the existing cumulative phase counters separately for WAL acceptance and local apply waiting. A real RF3 test pauses the applier after durable acceptance to verify the distinction; cancellation remains explicit. Full repository checks and CI passed; cloud latency attribution and throughput qualification remain separate.

  • Route mixed annotation HTTP requests across independent shard owners (#693), using stable turn routing for point reads and verified producer binding for updates, without re-encoding Arrow IPC. A real RF3 regression covers two ingresses, independent owners and owner replacement. Full repository checks and CI passed; deployment configuration and multi-worker throughput qualification remain separate.

  • Configure independent physical annotation shard applications (#694). Each owner has its own body/narrow tables and publication/recovery progress; shared ingresses use a validated route map. Real RF3/etcd process tests cover acknowledged suffix recovery while a peer drains. Full repository checks passed and all CI is green. Automatic client partitioning, migration and sustained multi-owner capacity remain separate.

  • Overlap independent annotation source metadata observations (#695). Both quorum results are validated before consuming a page; ordered application, bounded admission and closure fencing remain intact. A regression rejects a stale view without fetching or advancing. Full repository checks passed and all CI is green; no workload throughput gain is claimed.

  • Document annotation owner index and admission settings (#696). Each role consumes its own configuration; native ID maintenance belongs on the owner, with explicit prerequisites and coverage checks. Documentation only; all CI is green.

  • Reuse admitted annotation upload buffers during ordered apply (#697). Exact quorum-observed metadata selects the shared buffer after apply admission; cancellation, restart and remote submissions retain WAL fetch fallback. Durability, ordered apply and publication remain intact; all CI is green.

  • Expose annotation upload admission, body read, preparation, point lookup, and owner-authority timing in bounded diagnostic counters (#698). Existing admission, durability and consistency checks remain intact; all CI is green.

  • Expose bounded timing for annotation predecessor loading, target publication, selected-target validation, and durable catalog confirmation (#699). Existing publication ordering, fencing and recovery checks remain intact; all CI is green.

  • Exercise pipelined replication through independent TLS process leader failure, original-directory restart and interrupted snapshot recovery (#700). Both pipeline settings retain full payload and exact-request checks; runtime defaults and ACK durability remain unchanged; all CI is green.

  • Report whether ordered annotation apply is blocked by its memory budget, source window or staging capacity (#702). Preserve existing admission and publication behavior; observations are sampled states, not refusal counts or blocked durations; all CI is green.

  • Refuse HTTP/2 annotation uploads that lack memory before consuming their bodies, so waiting streams do not stall admitted uploads on the same connection (#703). Preserve existing memory budgets and explicit not-submitted outcomes; shared-connection regression and complete mixed-workload value/recovery checks cover the change; all CI is green.

  • Separate append RPC admission, connection, send and response timing from WAL execution timing (#705). Controlled RF3 waits/cancellation and a complete mixed-workload data/recovery audit validate the observation boundaries; no ACK, admission or retry policy changes; all CI is green.

  • Preserve committed coverage for each native scalar-index segment before combining matches (#706). Regression tests cover mutable status values, clearing to NULL, append-mode catch-up and reopening with stable/physical row IDs; this fixes the existing query wrapper and does not add an asynchronous indexing service; all CI is green.

  • Bound request-drain work before Raft progress with a configurable time budget (#707). Retain command batching, durable quorum/apply ACKs and recovery behavior; a gated RF3 regression checks progress and complete payloads after reopening; sustained mixed-workload qualification and all CI passed.

  • Opt-in bounded native status index maintenance (#708): idle catch-up, physical-byte merge admission, explicit empty-definition retirement, prepared-operation recovery with maintenance disabled, and retained-reader file reclamation. Filtering API/observable lag and status-inclusive mixed-load qualification remain separate work.

  • Bounded materialized status filters (#709): per-shard HTTP queries project IDs from one pinned Lance snapshot, include unindexed fragments, report materialization/index coverage, fence stale owners and retain GC pins through native query work. Active/frozen updates remain immediately point-readable; filters observe materialized state. No pagination or cross-shard snapshot.

Annotation applications now support explicit fresh physical shards, with one body/narrow table pair per owner and a complete ingress route map (#694). Configured routing and independent shard publication/recovery are implemented; the default remains one table and one virtual shard. Automatic session/client placement and migration, asynchronous status/label/score filtering indexes, and large-table index, storage and recovery costs remain open work. These explicit shard APIs do not establish saturation capacity or complete automatic scale-out.

Remaining qualification must run mixed new-body ingestion, point updates, cross-ingress reads, compaction and reclamation together; verify all accepted values and recovery progress; and report rejection rates, latency distributions, backlog/drain, process memory and storage costs. Separate-process and multi-host persistent-storage/blob tests must state their topology and fault boundary. Closed-loop load and small recovery fixtures must not be presented as fixed-arrival-rate capacity.

See the mutation contract, staging view, and owner runtime.

Lingua principale
Rust
Stelle
89
Fork
21
Merge medio
2h 6m
PR unite (30g)
435

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance-context

Tutte le issue di lance-format/lance-context

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.