Unify master maintenance scheduling (WAL merge / compaction / index / repair)
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 6/100
Direzione di ricerca
This is a multi-phase design tracker, not a starter task. Begin by reading docs/design/unified-maintenance-scheduler.md and docs/design/scheduler-improvement-backlog.md, then the eligibility logic in the claim_next path and merge/failure.rs. Only open P1 items with no PR are plausible for a newcomer, and only after confirming scope with maintainers, since the issue is still open with active discussion.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Goal
Replace the eleven independent scheduling, sweep and discovery loops in lance-context-master with one model: a single leader planner over a per-table demand table, per-(table, kind) scores with priority classes and aging, executor capacity published as (slots per kind, bytes), and one typed failure ledger. The merge-crate fence (target-locks, merge-executions, version barriers, progress watchdog) is not changed; it stays the correctness boundary.
Success criteria:
- One place answers "what runs next on table X and why" (
GET /api/v1/scheduler/demand?target=). - Scheduling decisions are pure functions of (demand, executors, fence state, ledger, policy, now) and unit-tested without etcd.
- Global scheduling env knobs ≤ 12 (today ≈ 71); per-table behaviour from one policy document (today six allowlists).
- No planner cycle issues a full prefix scan; one etcd txn per placement (today a 7-key snapshot per candidate plus a
running/page scan per claim). - Starvation cases covered by tests: continuous merger vs prepared compaction, low-count WAL tails, legacy RPCs holding merge slots, alphabet-late tables.
Why now
| Metric | Today |
|---|---|
| Independent scheduling/discovery loops | 11 |
| Pacing mechanisms | 4 (tokio interval, sleep, etcd next_batch_ms cursors, k8s Job lifecycle) |
| Failure/backoff systems | 5 |
| Admission surfaces | 3 (5 semaphores, drain gate, 7 etcd ownership keys) |
| etcd key families | 27 |
| Scheduling env knobs | ≈ 71 |
| Per-table allowlists | 6 |
| PRs 2026-09-30 → 10-08 fixing this layer | ~35 of 50 |
Queue keys are UUIDv7 (generate_id()), so order is approximately enqueue time; what is missing is any priority dimension.
Design highlights
Full design: docs/design/unified-maintenance-scheduler.md. Backlog with file:line references: docs/design/scheduler-improvement-backlog.md.
- Scale fit. Tens to low hundreds of active tables, 3–6 masters, tasks lasting seconds to hours. This is a multi-table storage-engine background scheduler (RocksDB column families, ClickHouse MergeTree, Pebble, Hudi table services), not a cluster scheduler. A single leader with a full in-memory view is sufficient; optimistic multi-claimer (what
claim_nextis today) and Omega-style shared state are rejected. - Demand table per table (
pending_generations,pending_bytes,oldest_pending_ms,fragment_count,index_stale,missing_fragments), fed by per-shard watermarks (sealed_through_seq, idempotent under loss/duplication) from writer flush and merge release, reconciled by the existing stats scan, which may lower a watermark only with a newer observed revision. Replaces merge sweep, wal-tail sweep, resident discovery, catch-up admission and the merge-requests loop. - Scores and classes.
merge_score = max(gens/min, bytes/min) × age_boost; classes Recovery > Critical merge (≥256 gens) > Commit-ready (generalises #324) > Normal merge > Compact/Index > Tail. Class order alone starves, so two hard bounds apply across classes:max_consecutive_turns[kind]per table andmax_turn_wait_secs[kind]promoting to class 1. - One write-turn holder per table = the existing merge-crate lock. Preparation (compaction/index) runs outside the turn, concurrently with the holder and with each other; the planner must never re-serialise merge behind preparation (#308, #327). Dependencies become demand post-conditions, not task DAG edges.
- Executors (resident master, worker, catch-up Job, legacy worker RPC) publish leased heartbeats with
slots[kind];bytes_freeis a sample only. Each assignment carriesreserved_bytes, headroom =bytes_total − Σ reservedover live assignments (rebuilt from etcd on failover), and executors still enforce the local memory budget. Placements are rate-limited per executor (TiKV PD store-limit). k8s Jobs become an executor type, not a separate controller. Isolating legacy RPCs bounds blast radius only; stall detection stays on the execution's actual read/encode/commit progress. - Typed errors with stable codes replace message-text classification (#328 matches on the
store_base.rs:path; a path move or Lance wrapper change breaks it, line renumbering does not). - Proportional backpressure: per-table
backlog_classso writers slow before the 4096-generation 503 cliff. - Single-leader risks (hot spot, split brain on lease expiry, stale state, thundering herd) and their mitigations are listed in design §5.
Plan
- P0 — no behaviour change
- Design doc + backlog doc (#334, #335)
- Extract
eligibility()pure function fromclaim_next, unit tests, skip-reason logging (#336) - Stable
MaintenanceErrorCode; classify by code inmerge/failure.rs(#338) - Metrics:
schedule_to_bind_seconds,turn_wait_seconds{kind}, commit-ready served/expired, resident pool depth (#339) - Slow-but-alive index build e2e test (#327 + #330 + #328 interaction) (#340)
-
/merge-progress: single-revision read, typed response (#341) - Operator runbook (#342)
- P1 — demand table + shadow planner (every existing loop keeps running unchanged; shadow places nothing and removes nothing)
- P1 data model spec (#343)
-
TableDemand::foldpure function with permutation/idempotency tests (#344) - Writers publish per-shard
demand-eventswatermarks on every flush (#345, detached in #351) - Executors publish
merged_throughin the merge-crate release txn (#350) - Stats scanner additionally writes
demand-events(sourcescan) (#356) - Fold is a per-dimension join ordered by generation; rebuild-equals-fold property test (#359, replaced #352 after review rounds 3–4)
- Merged watermarks published after release, one put per shard (#357)
- Leader election + planner folds
demand-events/intoP/demand/;GET /scheduler/demand(#361) - Leader election; planner logs intended placements;
GET /scheduler/demand - Executor heartbeats; headroom from durable
reserved_bytes(#362) - Scores + classes + oldest-first class 1; shadow placements;
scheduler_shadow_disagreements_total(#363)
- P2 — planner owns MergeWal for
mode: plannertables, switched per table only once every master runs a policy-aware binary; legacy workers become an executor kind; delete resident-only pool (#331), wal-tail sweep, resident discovery for migrated tables - P3 — Compact / IndexId / Repair / Retire; catch-up Jobs as executor type
- P4 — cleanup: delete pool pollers, multi-key claim, cursor loops, allowlists, ~55 env knobs, 15 etcd key families; split
task_store.rs
Mixed-version rule for every phase: old masters ignore unknown key families and keep honouring target-locks/merge-executions; no key family is deleted while any binary reads it.
Open questions before P1
- Production table count and hot-table fraction.
- Generation size distribution (bytes/rows).
- p99 compaction and index-build duration on the largest tables.
- Can the rollout server coalesce generations under
criticalbacklog, or must backpressure stay read-side?
PRs
#334 design · #335 backlog · #336 eligibility module · #337 review fixes · #338 error code · #339 metrics · #340 watchdog test · #341 merge-progress · #342 runbook · #343 P1 spec · #344 demand fold · #345 writer demand events · #347 #348 #349 review-2 fixes · #350 merged watermarks in release · #351 #352 #353 #354 #355 review-3 fixes · #356 scan demand · #357 #358 #359 #360 review-4 fixes · #355 real-IO watchdog test · #361 #362 #363 P1.5–1.7 planner · #364 #365 review-5 fixes · #366 review-6 fixes · #367 #368 review-7 fixes + canary runbook · #371 failover + isolation tests
- Lingua principale
- Rust
- Stelle
- 89
- Fork
- 21
- Merge medio
- 2h 6m
- PR unite (30g)
- 435
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di lance-format/lance-context
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 20/100
lance-format/lance-context#832 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
lance-format/lance-context#378 · 119 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
lance-format/lance-context#260 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 28/100
lance-format/lance-context#214 ·
I maintainer di solito rispondono entro 1 giorno
-
RolloutStore::dataset should be interior-mutable so merge/compact can refresh the handle without an exclusive lockForse già presa @lucyge2022 l’ha presa 97 giorni fa. Apertaenhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 42/100
lance-format/lance-context#198 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di lance-format/lance-context
Issue simili
-
comp-rbac
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ClickHouse/ClickHouse#131875 ·
I maintainer di solito rispondono entro 1 giorno
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesForse già presa @Borg93 l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#11117 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Forse già presa @Ynyesto l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
foundry-rs/foundry#18485 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
bondybondbond/rushcut#238 · 1 commento ·