Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

bottomless: finished S3 upload tasks are never reaped, so sqld memory grows by ~10 KB per WAL batch

Aperta Adatta ai principianti
  • Qualcosa non va?
#2,298 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

@pavanmanishd ci sta già lavorando.

Dal 22/10/2026.

  • #2299 di @pavanmanishd — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
83/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
aws, docker, rust
Ambito
backend, databases

Direzione di ricerca

Inizia da bottomless/src/replicator.rs e segui il ciclo di caricamento su S3 in Replicator::with_options; leggi come JoinSet gestisce le attività completate mentre riceve le richieste. L’issue propone di recuperare le attività completate all’interno di quel ciclo. Conferma la modifica riproducendo caricamenti WAL continui e verificando che la memoria non cresca più a ogni batch.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

With bottomless replication enabled, sqld's memory grows linearly with the number of WAL batches it uploads and never comes back down. Every uploaded batch leaves about 10 KB behind. A database with a small but steady write stream (one write per second, default 1 s batch interval) grows by roughly 80–130 MB per day until the process is restarted or OOM-killed.

This may be the same report as #1781. It is separate from the snapshot-upload defects in #2260, which does not change the WAL upload loop.

Cause

In bottomless/src/replicator.rs, Replicator::with_options spawns the S3 upload loop. The loop creates an inner JoinSet and spawns one task per upload request:

let mut join_set = JoinSet::new();
while let Some(req) = frames_inbox.recv().await {
    // ...
    join_set.spawn(async move { /* put_object, remove_file, update progress */ });
}
while join_set.join_next().await.is_some() {}

Nothing reads the results of finished tasks until frames_inbox closes, which happens only at shutdown. A JoinSet keeps every finished task (its allocation and output) until it is joined, so each uploaded batch stays in memory for the lifetime of the replicator. new_generation reuses the same replicator, so the set is not reset at checkpoints either.

Fix

Reap finished tasks inside the loop:

                 while let Some(req) = frames_inbox.recv().await {
+                    while join_set.try_join_next().is_some() {}
                     tracing::trace!("Received S3 upload request: {}", req.path);

Reproducer

Local Docker, ghcr.io/tursodatabase/libsql-server:361d082 (same code in d6c75af / v0.24.33), S3-compatible store (RustFS) on the same network.

sqld environment (both containers):

SQLD_ENABLE_BOTTOMLESS_REPLICATION=true
LIBSQL_BOTTOMLESS_ENDPOINT=http://s3:9000
LIBSQL_BOTTOMLESS_BUCKET=sqld-leak
LIBSQL_BOTTOMLESS_DATABASE_ID=<unique per container>
LIBSQL_BOTTOMLESS_BATCH_INTERVAL_SECS=1
SQLD_CHECKPOINT_INTERVAL_S=60
LIBSQL_DISABLE_METRICS=true

Load: 10 INSERT OR REPLACE statements per second per container over HTTP. Memory: anon from the container's memory.stat.

45 min, 10 writes/s anon growth, minute 16 → 38 per uploaded batch
361d082, batch every 1 s 29.2 → 43.1 MB 10.8 KB
d6c75af, batch every 1 s same slope 11.2 KB
361d082, batch every 60 s flattens —
361d082 + fix above, batch every 1 s 18.2 → 21.8 MB 2.8 KB (see below)
bottomless off 6.3 → 6.6 MB —

Growth tracks the number of uploaded batches, not the number of writes: the same write load with a 60 s batch interval stays nearly flat.

Remaining growth with the fix

With the fix, about 2.8 KB per batch still accumulates. A 15-minute rheaper heap profile (/profile/heap/enable) of the patched build shows the live Rust heap flat at ≈ 0.25 MB and only 0.17 MB of allocations never freed, with no growing call stack. So the remainder is not a Rust-side leak. It is glibc keeping freed memory in per-thread arenas: the same patched build with MALLOC_ARENA_MAX=2 stays flat at 13.5 MB from minute 18 to minute 41 (1,380 batches), while the patched build with default glibc settings rises 17.0 → 21.2 MB over the same window.

Environment

  • libsql-server 361d082 and d6c75af (official images), linux/arm64 under Docker Desktop; same growth pattern observed on linux/amd64 in production.
  • Allocator: glibc (system).
Lingua principale
C
Stelle
17.3k
Fork
545
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tursodatabase/libsql

Tutte le issue di tursodatabase/libsql

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.