SeqnosByKey_Bucketed writes an empty file per key for every bucket since seqno 0

Abierto
  • ¿Algo no está bien?
#8,673 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
databases, performance

Línea de trabajo

Start with src/indexing/strategies/seqnos_by_key_bucketed.cpp, especially visit_entry, store_empty_buckets, query_bucket, and current_results; the issue links directly to these sections. Trace how missing buckets are distinguished from empty ones and how queries use current_results. Done means sparse keys no longer generate or fetch empty buckets, queries still detect lost data correctly, and per-key memory use is bounded; the issue presents design ideas but leaves the approach open.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

SeqnosByKey_Bucketed stores a file per key for every bucket from seqno 0 up to the key's current bucket, including empty ones:

  • The first time it sees a key at seqno S, it stores an empty bucket for every bucket before S (visit_entry, store_empty_buckets). With 1,000 seqnos per bucket, a key first written at seqno 5,000,000 costs 5,000 files.
  • When a key is written again after a gap, it stores an empty bucket for every bucket in the gap (L384-L388).

So the index costs about keys × seqno / bucket size files, however few writes each key has. The empty files are needed today because a query treats a missing bucket as lost data and re-indexes from seqno 0 (L254-L283).

Separately, current_results keeps an entry for every key ever indexed and never frees it (L24-L26). Queries depend on that, since a key without an entry is treated as never written (L311-L322), so entries can't just be evicted.

Each node builds all of this on its own, and again after every restart, because .index is wiped on start (enclave_lfs_access.h#L295-L302).

We use two LazyStrategy<SeqnosByKey_Bucketed<...>> indexes (1,000 seqnos per bucket, 200 buckets) keyed by collection and by (collection, tag) and we see the following:

  • In a 3-node sandbox on 7.0.23, building the index at keys × seqno ≈ 1.6e9 wrote 789k files: 3.3 GB on disk (about 4.2 KB each). Node RSS went up about 323 MiB and didn't come back after the build. We didn't dig into the RSS part. The LFS keeps no per-file state, so it may be the queued async writes plus allocator retention.
  • On one production instance, CCF logged 182k Writing LFS file operations slower than 10 ms over three weeks, up to 4.3 s each. All of them were 108-116 byte files, so empty or single-entry buckets.
  • Range queries on sparse keys also fetch every empty bucket in the range, which churns the shared bucket LRU.

To Reproduce

A standalone harness that calls visit_entry and get_write_txs_impl on the real strategy (main at 00d0976e7, 1,000 seqnos per bucket, 200 buckets), with an in-memory LFS that counts store() calls:

  • Key A first written at seqno 1: no files.
  • Key B first written at seqno 1,000,000: 1,000 files, all empty.
  • Key A written again at seqno 2,000,000: 2,000 files (1 with data, 1,999 empty).
  • 100 new keys first written around seqno 5,000,000: 500,000 empty files.
  • A query for B over the 198,000 seqnos before its first write: 198 bucket fetches, all empty.
  • 200,000 distinct keys: 200,000 current_results entries (about 160 B each with 21-byte keys), still there after later writes move the watermark on.
  • With an LFS that drops empty buckets, the next query on a range in the gap resets the watermark to 0 and clears every key.

Happy to share the harness used for checking this.

Expected behavior

Index cost should grow with the number of writes, not with keys × ledger length, and per-key memory should be bounded. A couple of ideas, but you know this code better:

  • Don't store empty buckets. Have each stored bucket, and the in-memory current one, record the start of the key's previous non-empty bucket, so a query can tell an empty bucket from a lost file and only fetches buckets with data.
  • Flush and drop current_results entries for keys that haven't been written for a while, keeping a small marker on disk so queries still know the key exists.

Environment information

CCF 7.0.23. The code is unchanged on main at 00d0976e71afc0613328225e6f4074a78f52a592.

Additional context

An app can't bound this index by key count alone, since a key created late in a ledger's life costs seqno / bucket size files. The workaround we're looking at is turning the index off above a keys × seqno threshold, but that makes range queries on exactly those ledgers about 2x slower.

Lenguaje dominante
C++
Estrellas
876
Forks
264
Merge medio
1 d 13 h
PR fusionados (30 d)
116

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/CCF

Todos los issues de microsoft/CCF

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.