Test worker loss, expose Ballista retry settings and handle SIGTERM in scheduler and worker

Abierto
  • ¿Algo no está bien?
#31 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
distributed-systems, testing

Línea de trabajo

Start with crates/oxidelake-runtime/tests/cluster.rs:109-132 and tests/cli.rs:202-280 to see how the cluster tests launch queries, then read oxide_scheduler.rs and bin/oxide_worker.rs at the cited entry points. Check how Ballista 54 configures retries, worker shutdown and health reporting. Done means the Linux worker loss test, scheduler retry flags, signal handling and drain timeout are implemented and the documented or added health endpoint is verified.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Today — the cluster tests prove equality of results, not resilience. crates/oxidelake-runtime/tests/cluster.rs:109-132 starts an in-process scheduler + 2 executors and runs 4 queries; tests/cli.rs:202-280 spawns one scheduler + one worker. Neither kills an executor, restarts the scheduler or checks a failed task. SchedulerConfig::default() is spread with task_max_failures/stage_max_failures not surfaced (oxide_scheduler.rs has three flags: bind-host, port, cluster-backend). bin/oxide_worker.rs:283-301 and oxide_scheduler.rs:235-248 are #[tokio::main] bodies that .await Ballista and return: no signal handler, no drain, no readiness signal of OxideLake's own.

Why it is worth fixing — "what happens on worker loss" is whatever Ballista 54 does by default, and nothing in this repository proves it; roadmap Phase 5's "clean shutdown" is asserted only by the test process exiting.

Fix —

  • An integration test that starts two oxide-worker processes, kills one mid-query (a slow query over the 11 MiB fixture), and asserts the query completes with the same result or fails with a clear error naming the lost executor.
  • --task-max-failures / --stage-max-failures flags on the scheduler, passed into SchedulerConfig.
  • tokio::signal handling for SIGTERM/SIGINT in both binaries: stop accepting tasks, wait for in-flight tasks up to a --drain-timeout, exit 0; document Ballista's REST/health endpoints or add /healthz.

Done when — the worker-loss test runs in the Linux lane; kill -TERM on a worker returns 0 within the drain timeout (tested); the flags appear in --help.

Lenguaje dominante
Rust
Estrellas
1
Forks
2
Merge medio
1 d 11 h
PR fusionados (30 d)
7

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de vyncint/oxidelake

Todos los issues de vyncint/oxidelake

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.