Test worker loss, expose Ballista retry settings and handle SIGTERM in scheduler and worker

Aperta
  • Qualcosa non va?
#31 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
distributed-systems, testing

Direzione di ricerca

Start with crates/oxidelake-runtime/tests/cluster.rs:109-132 and tests/cli.rs:202-280 to see how the cluster tests launch queries, then read oxide_scheduler.rs and bin/oxide_worker.rs at the cited entry points. Check how Ballista 54 configures retries, worker shutdown and health reporting. Done means the Linux worker loss test, scheduler retry flags, signal handling and drain timeout are implemented and the documented or added health endpoint is verified.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Today — the cluster tests prove equality of results, not resilience. crates/oxidelake-runtime/tests/cluster.rs:109-132 starts an in-process scheduler + 2 executors and runs 4 queries; tests/cli.rs:202-280 spawns one scheduler + one worker. Neither kills an executor, restarts the scheduler or checks a failed task. SchedulerConfig::default() is spread with task_max_failures/stage_max_failures not surfaced (oxide_scheduler.rs has three flags: bind-host, port, cluster-backend). bin/oxide_worker.rs:283-301 and oxide_scheduler.rs:235-248 are #[tokio::main] bodies that .await Ballista and return: no signal handler, no drain, no readiness signal of OxideLake's own.

Why it is worth fixing — "what happens on worker loss" is whatever Ballista 54 does by default, and nothing in this repository proves it; roadmap Phase 5's "clean shutdown" is asserted only by the test process exiting.

Fix —

  • An integration test that starts two oxide-worker processes, kills one mid-query (a slow query over the 11 MiB fixture), and asserts the query completes with the same result or fails with a clear error naming the lost executor.
  • --task-max-failures / --stage-max-failures flags on the scheduler, passed into SchedulerConfig.
  • tokio::signal handling for SIGTERM/SIGINT in both binaries: stop accepting tasks, wait for in-flight tasks up to a --drain-timeout, exit 0; document Ballista's REST/health endpoints or add /healthz.

Done when — the worker-loss test runs in the Linux lane; kill -TERM on a worker returns 0 within the drain timeout (tested); the flags appear in --help.

Lingua principale
Rust
Stelle
1
Fork
2
Merge medio
1g 11h
PR unite (30g)
7

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di vyncint/oxidelake

Tutte le issue di vyncint/oxidelake

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.