Unify master maintenance scheduling (WAL merge / compaction / index / repair)

Aperta
#333 11 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
6/100
Tipo di issue
Refactoring
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend

Direzione di ricerca

This is a multi-phase design tracker, not a starter task. Begin by reading docs/design/unified-maintenance-scheduler.md and docs/design/scheduler-improvement-backlog.md, then the eligibility logic in the claim_next path and merge/failure.rs. Only open P1 items with no PR are plausible for a newcomer, and only after confirming scope with maintainers, since the issue is still open with active discussion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Goal

Replace the eleven independent scheduling, sweep and discovery loops in lance-context-master with one model: a single leader planner over a per-table demand table, per-(table, kind) scores with priority classes and aging, executor capacity published as (slots per kind, bytes), and one typed failure ledger. The merge-crate fence (target-locks, merge-executions, version barriers, progress watchdog) is not changed; it stays the correctness boundary.

Success criteria:

  • One place answers "what runs next on table X and why" (GET /api/v1/scheduler/demand?target=).
  • Scheduling decisions are pure functions of (demand, executors, fence state, ledger, policy, now) and unit-tested without etcd.
  • Global scheduling env knobs ≤ 12 (today ≈ 71); per-table behaviour from one policy document (today six allowlists).
  • No planner cycle issues a full prefix scan; one etcd txn per placement (today a 7-key snapshot per candidate plus a running/ page scan per claim).
  • Starvation cases covered by tests: continuous merger vs prepared compaction, low-count WAL tails, legacy RPCs holding merge slots, alphabet-late tables.

Why now

Metric Today
Independent scheduling/discovery loops 11
Pacing mechanisms 4 (tokio interval, sleep, etcd next_batch_ms cursors, k8s Job lifecycle)
Failure/backoff systems 5
Admission surfaces 3 (5 semaphores, drain gate, 7 etcd ownership keys)
etcd key families 27
Scheduling env knobs ≈ 71
Per-table allowlists 6
PRs 2026-09-30 → 10-08 fixing this layer ~35 of 50

Queue keys are UUIDv7 (generate_id()), so order is approximately enqueue time; what is missing is any priority dimension.

Design highlights

Full design: docs/design/unified-maintenance-scheduler.md. Backlog with file:line references: docs/design/scheduler-improvement-backlog.md.

  • Scale fit. Tens to low hundreds of active tables, 3–6 masters, tasks lasting seconds to hours. This is a multi-table storage-engine background scheduler (RocksDB column families, ClickHouse MergeTree, Pebble, Hudi table services), not a cluster scheduler. A single leader with a full in-memory view is sufficient; optimistic multi-claimer (what claim_next is today) and Omega-style shared state are rejected.
  • Demand table per table (pending_generations, pending_bytes, oldest_pending_ms, fragment_count, index_stale, missing_fragments), fed by per-shard watermarks (sealed_through_seq, idempotent under loss/duplication) from writer flush and merge release, reconciled by the existing stats scan, which may lower a watermark only with a newer observed revision. Replaces merge sweep, wal-tail sweep, resident discovery, catch-up admission and the merge-requests loop.
  • Scores and classes. merge_score = max(gens/min, bytes/min) × age_boost; classes Recovery > Critical merge (≥256 gens) > Commit-ready (generalises #324) > Normal merge > Compact/Index > Tail. Class order alone starves, so two hard bounds apply across classes: max_consecutive_turns[kind] per table and max_turn_wait_secs[kind] promoting to class 1.
  • One write-turn holder per table = the existing merge-crate lock. Preparation (compaction/index) runs outside the turn, concurrently with the holder and with each other; the planner must never re-serialise merge behind preparation (#308, #327). Dependencies become demand post-conditions, not task DAG edges.
  • Executors (resident master, worker, catch-up Job, legacy worker RPC) publish leased heartbeats with slots[kind]; bytes_free is a sample only. Each assignment carries reserved_bytes, headroom = bytes_total − Σ reserved over live assignments (rebuilt from etcd on failover), and executors still enforce the local memory budget. Placements are rate-limited per executor (TiKV PD store-limit). k8s Jobs become an executor type, not a separate controller. Isolating legacy RPCs bounds blast radius only; stall detection stays on the execution's actual read/encode/commit progress.
  • Typed errors with stable codes replace message-text classification (#328 matches on the store_base.rs: path; a path move or Lance wrapper change breaks it, line renumbering does not).
  • Proportional backpressure: per-table backlog_class so writers slow before the 4096-generation 503 cliff.
  • Single-leader risks (hot spot, split brain on lease expiry, stale state, thundering herd) and their mitigations are listed in design §5.

Plan

  • P0 — no behaviour change
    • Design doc + backlog doc (#334, #335)
    • Extract eligibility() pure function from claim_next, unit tests, skip-reason logging (#336)
    • Stable MaintenanceErrorCode; classify by code in merge/failure.rs (#338)
    • Metrics: schedule_to_bind_seconds, turn_wait_seconds{kind}, commit-ready served/expired, resident pool depth (#339)
    • Slow-but-alive index build e2e test (#327 + #330 + #328 interaction) (#340)
    • /merge-progress: single-revision read, typed response (#341)
    • Operator runbook (#342)
  • P1 — demand table + shadow planner (every existing loop keeps running unchanged; shadow places nothing and removes nothing)
    • P1 data model spec (#343)
    • TableDemand::fold pure function with permutation/idempotency tests (#344)
    • Writers publish per-shard demand-events watermarks on every flush (#345, detached in #351)
    • Executors publish merged_through in the merge-crate release txn (#350)
    • Stats scanner additionally writes demand-events (source scan) (#356)
    • Fold is a per-dimension join ordered by generation; rebuild-equals-fold property test (#359, replaced #352 after review rounds 3–4)
    • Merged watermarks published after release, one put per shard (#357)
    • Leader election + planner folds demand-events/ into P/demand/; GET /scheduler/demand (#361)
    • Leader election; planner logs intended placements; GET /scheduler/demand
    • Executor heartbeats; headroom from durable reserved_bytes (#362)
    • Scores + classes + oldest-first class 1; shadow placements; scheduler_shadow_disagreements_total (#363)
  • P2 — planner owns MergeWal for mode: planner tables, switched per table only once every master runs a policy-aware binary; legacy workers become an executor kind; delete resident-only pool (#331), wal-tail sweep, resident discovery for migrated tables
  • P3 — Compact / IndexId / Repair / Retire; catch-up Jobs as executor type
  • P4 — cleanup: delete pool pollers, multi-key claim, cursor loops, allowlists, ~55 env knobs, 15 etcd key families; split task_store.rs

Mixed-version rule for every phase: old masters ignore unknown key families and keep honouring target-locks/merge-executions; no key family is deleted while any binary reads it.

Open questions before P1

  1. Production table count and hot-table fraction.
  2. Generation size distribution (bytes/rows).
  3. p99 compaction and index-build duration on the largest tables.
  4. Can the rollout server coalesce generations under critical backlog, or must backpressure stay read-side?

PRs

#334 design · #335 backlog · #336 eligibility module · #337 review fixes · #338 error code · #339 metrics · #340 watchdog test · #341 merge-progress · #342 runbook · #343 P1 spec · #344 demand fold · #345 writer demand events · #347 #348 #349 review-2 fixes · #350 merged watermarks in release · #351 #352 #353 #354 #355 review-3 fixes · #356 scan demand · #357 #358 #359 #360 review-4 fixes · #355 real-IO watchdog test · #361 #362 #363 P1.5–1.7 planner · #364 #365 review-5 fixes · #366 review-6 fixes · #367 #368 review-7 fixes + canary runbook · #371 failover + isolation tests

Lingua principale
Rust
Stelle
89
Fork
21
Merge medio
2h 6m
PR unite (30g)
435

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance-context

Tutte le issue di lance-format/lance-context

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.