Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

bottomless: finished S3 upload tasks are never reaped, so sqld memory grows by ~10 KB per WAL batch

Abierto Apto para principiantes
  • ¿Algo no está bien?
#2,298 1 comentario 0 reacciones 0 asignados Ver en GitHub

@pavanmanishd ya está trabajando en esto.

Desde el 22/10/2026.

  • #2299 de @pavanmanishd — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
83/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
aws, docker, rust
Área
backend, databases

Línea de trabajo

Empieza en bottomless/src/replicator.rs y sigue el bucle de subida a S3 en Replicator::with_options; lee cómo JoinSet gestiona las tareas completadas mientras recibe solicitudes. El issue propone recoger las tareas completadas dentro de ese bucle. Confirma el cambio reproduciendo subidas continuas de WAL y comprobando que la memoria ya no crece con cada lote.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

With bottomless replication enabled, sqld's memory grows linearly with the number of WAL batches it uploads and never comes back down. Every uploaded batch leaves about 10 KB behind. A database with a small but steady write stream (one write per second, default 1 s batch interval) grows by roughly 80–130 MB per day until the process is restarted or OOM-killed.

This may be the same report as #1781. It is separate from the snapshot-upload defects in #2260, which does not change the WAL upload loop.

Cause

In bottomless/src/replicator.rs, Replicator::with_options spawns the S3 upload loop. The loop creates an inner JoinSet and spawns one task per upload request:

let mut join_set = JoinSet::new();
while let Some(req) = frames_inbox.recv().await {
    // ...
    join_set.spawn(async move { /* put_object, remove_file, update progress */ });
}
while join_set.join_next().await.is_some() {}

Nothing reads the results of finished tasks until frames_inbox closes, which happens only at shutdown. A JoinSet keeps every finished task (its allocation and output) until it is joined, so each uploaded batch stays in memory for the lifetime of the replicator. new_generation reuses the same replicator, so the set is not reset at checkpoints either.

Fix

Reap finished tasks inside the loop:

                 while let Some(req) = frames_inbox.recv().await {
+                    while join_set.try_join_next().is_some() {}
                     tracing::trace!("Received S3 upload request: {}", req.path);

Reproducer

Local Docker, ghcr.io/tursodatabase/libsql-server:361d082 (same code in d6c75af / v0.24.33), S3-compatible store (RustFS) on the same network.

sqld environment (both containers):

SQLD_ENABLE_BOTTOMLESS_REPLICATION=true
LIBSQL_BOTTOMLESS_ENDPOINT=http://s3:9000
LIBSQL_BOTTOMLESS_BUCKET=sqld-leak
LIBSQL_BOTTOMLESS_DATABASE_ID=<unique per container>
LIBSQL_BOTTOMLESS_BATCH_INTERVAL_SECS=1
SQLD_CHECKPOINT_INTERVAL_S=60
LIBSQL_DISABLE_METRICS=true

Load: 10 INSERT OR REPLACE statements per second per container over HTTP. Memory: anon from the container's memory.stat.

45 min, 10 writes/s anon growth, minute 16 → 38 per uploaded batch
361d082, batch every 1 s 29.2 → 43.1 MB 10.8 KB
d6c75af, batch every 1 s same slope 11.2 KB
361d082, batch every 60 s flattens —
361d082 + fix above, batch every 1 s 18.2 → 21.8 MB 2.8 KB (see below)
bottomless off 6.3 → 6.6 MB —

Growth tracks the number of uploaded batches, not the number of writes: the same write load with a 60 s batch interval stays nearly flat.

Remaining growth with the fix

With the fix, about 2.8 KB per batch still accumulates. A 15-minute rheaper heap profile (/profile/heap/enable) of the patched build shows the live Rust heap flat at ≈ 0.25 MB and only 0.17 MB of allocations never freed, with no growing call stack. So the remainder is not a Rust-side leak. It is glibc keeping freed memory in per-thread arenas: the same patched build with MALLOC_ARENA_MAX=2 stays flat at 13.5 MB from minute 18 to minute 41 (1,380 batches), while the patched build with default glibc settings rises 17.0 → 21.2 MB over the same window.

Environment

  • libsql-server 361d082 and d6c75af (official images), linux/arm64 under Docker Desktop; same growth pattern observed on linux/amd64 in production.
  • Allocator: glibc (system).
Lenguaje dominante
C
Estrellas
17.3k
Forks
545
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de tursodatabase/libsql

Todos los issues de tursodatabase/libsql

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.