RocksDB: dead-workflow backfill materializes unbounded range before batch cutoff

Abierto
  • ¿Algo no está bien?
#5,701 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@brittianwarner ya está trabajando en esto.

Desde el 11/9/2026.

  • #5707 de @brittianwarner — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend, databases

Línea de trabajo

Comienza con gasoline/src/db/kv/debug.rs, especialmente con backfill_dead_workflows y su límite MAX_BACKFILLS, y luego sigue el manejo de rangos a través de universaldb/src/range_option.rs y los archivos de transacciones de RocksDB y transaction_task.rs. Usa el escenario de prueba propuesto para filesystem/RocksDB para comprobar el tamaño de la obtención materializada, la agregación parcial de workflows, la indexación de workflows muertos y la continuación entre páginas o después de un reinicio. Hecho significa obtener datos con memoria acotada, sin clasificaciones erróneas ni progreso detenido.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

In Rivet engine 2.3.17, gasoline_dead_wf_backfill requests the entire remaining workflow-data range without a row limit. The RocksDB adapter materializes that range before yielding the first entry. Consequently, the backfill's MAX_BACKFILLS = 1000 cutoff does not bound the underlying read allocation.

This report is based on public source inspection. It does not include a completed minimal runtime reproduction or claim a measured OOM threshold.

Affected code path

  1. Backfill workflow declares MAX_BACKFILLS = 1000, runs a chunk, and persists its continuation after the activity returns.
  2. backfill_dead_workflows scans the entire remaining workflow-data subspace using StreamingMode::WantAll, with no explicit range limit.
  3. The default range limit is None.
  4. The RocksDB stream adapter awaits the complete get_range result and collects a vector before yielding entries.
  5. The RocksDB transaction task uses limit.unwrap_or(usize::MAX) and copies matching keys and values into a vector.
  6. The backfill's workflow-count cutoff executes only after the underlying range has been materialized.

Lowering MAX_BACKFILLS alone therefore does not bound the initial allocation. Subsequent chunks can also re-materialize the remaining range. A sufficiently large workflow-data subspace could cause high memory consumption or OOM during this migration.

Introduction

The backfill and its startup dispatch were introduced by #5656, commit b92d85a820e00c41054c2804587d673fcde6ea1f (chore: sync engine changes from enterprise), listed in the 2.3.15 release notes. The backfill file is absent in 2.3.13 and this code path remains in 2.3.17.

Expected behavior

Memory consumption should be bounded by a page/batch budget rather than the size of the entire remaining workflow-data range. Progress should survive restart, and each workflow should only be classified after all required fields have been read.

Proposed reproduction / regression test

This is a suggested synthetic test, not a completed reproduction:

  1. Populate a filesystem/RocksDB-backed test store with many workflow-data records and substantial synthetic payloads.
  2. Leave the gasoline_dead_wf completion marker unset, as on an upgrade from 2.3.13.
  3. Run the backfill while measuring peak memory and the number/bytes of key-value pairs materialized by each range fetch.
  4. Check that the first fetch currently materializes the remaining range despite the 1,000-workflow processing cutoff.
  5. After a fix, assert bounded fetches, correct dead-workflow indexing, and continuation correctness when one workflow spans multiple pages or execution restarts between pages.

Possible fix

Either paginate this backfill explicitly or implement genuinely bounded pagination in the RocksDB range-stream adapter. Preserve partial workflow aggregation across page boundaries (or an equivalent safe continuation scheme) so a row limit does not misclassify workflows or repeatedly rescan a single oversized workflow without making progress. A broader adapter change would need transaction/snapshot-semantics coverage.

Lenguaje dominante
Rust
Estrellas
6.5k
Forks
291
Merge medio
16 h 58 min
PR fusionados (30 d)
33

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de rivet-dev/rivet

Todos los issues de rivet-dev/rivet

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.