Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

perf(query): narrow glob pruning, cache footprint and trace-candidate reads

Aperta
  • Qualcosa non va?
#281 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
databases, observability, performance

Direzione di ricerca

Start with internal/query/file_snapshot.go and the review threads linked from issue #281, then trace the read_trace_parts and read_trace_candidates paths described in the issue. Measure glob file reads, cache size and trace-read cost under the stated workloads before choosing an approach. Done means the four review follow-ups are addressed, parity coverage is added where requested, and the measured trade-offs are documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

PR #272 review follow-ups from the e818a8db review (https://github.com/labstack/fanout/pull/272#pullrequestreview-5385883605). None of these affect correctness. Each item below links its original thread.

  1. Captured-ID glob filter does not prune files (medium, measured): https://github.com/labstack/fanout/pull/272#discussion_r4160763194
    internal/query/file_snapshot.go filters the broad-selection glob with substr(filename, …) IN (ids). DuckDB's multi-file reader does not push that down, so every file matching the glob is opened and scanned. With 100 batches and 80 selected, profiles show 101 files read (explicit list: 80; filename IN ('<full path>', …): 81). This path also loses newest-first ordering for top-N reads. Options: filter on full paths, or raise the explicit-list threshold, and measure statement size against files read.

  2. Read-cache catalog is about as large as the Parquet data (medium-low, measured): https://github.com/labstack/fanout/pull/272#discussion_r4160763216
    After a 300 s run at 120k offered rows/s (telemetrygen, two spans per trace), catalog.duckdb was 762 MiB against 529 MiB of Parquet: about 474 MiB of used blocks and 288 MiB free. read_trace_parts and read_trace_candidates each held about 3.6M rows. Document that cache size scales with trace count and that free blocks are not returned without a rebuild, and consider compaction of the cache file.

  3. Trace candidates re-aggregate every overlapping part when any cached file straddles the window (low, plausible): https://github.com/labstack/fanout/pull/272#discussion_r4160763233
    Under continuous ingest the newest cached file usually straddles the window end, so most trace reads group read_trace_parts for all overlapping batches instead of reading read_trace_candidates. Restrict the parts aggregation to trace_id IN (partial) and take fully contained traces from the index, as the other branch does. Add a parity test and measure trace read cost as retention grows.

  4. The candidate index is evaluated twice in the contained-files branch (low): https://github.com/labstack/fanout/pull/272#discussion_r4160763241
    The branch added in e7dbc79a reads index once for the fully contained rows and again inside partial. On scoped reads with mixed-scope traces this repeats the read_trace_parts union. Results are unaffected. Bind it once, or share it with the rewrite in item 3.

Related: #277 (ingest CPU per row), #278 (sustained read capacity and peak RSS).

Lingua principale
HTML
Stelle
6
Fork
0
Merge medio
2h 25m
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di labstack/fanout

Tutte le issue di labstack/fanout

Issue simili

Altre issue su Databases

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.