Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

perf(query): narrow glob pruning, cache footprint and trace-candidate reads

オープン
  • 何か問題がありますか？
#281 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
活発
技術スタック
go
領域
databases, observability, performance

調査の方向性

Start with internal/query/file_snapshot.go and the review threads linked from issue #281, then trace the read_trace_parts and read_trace_candidates paths described in the issue. Measure glob file reads, cache size and trace-read cost under the stated workloads before choosing an approach. Done means the four review follow-ups are addressed, parity coverage is added where requested, and the measured trade-offs are documented.

索引モデルが issue の本文から書いたものです。

説明

PR #272 review follow-ups from the e818a8db review (https://github.com/labstack/fanout/pull/272#pullrequestreview-5385883605). None of these affect correctness. Each item below links its original thread.

  1. Captured-ID glob filter does not prune files (medium, measured): https://github.com/labstack/fanout/pull/272#discussion_r4160763194
    internal/query/file_snapshot.go filters the broad-selection glob with substr(filename, …) IN (ids). DuckDB's multi-file reader does not push that down, so every file matching the glob is opened and scanned. With 100 batches and 80 selected, profiles show 101 files read (explicit list: 80; filename IN ('<full path>', …): 81). This path also loses newest-first ordering for top-N reads. Options: filter on full paths, or raise the explicit-list threshold, and measure statement size against files read.

  2. Read-cache catalog is about as large as the Parquet data (medium-low, measured): https://github.com/labstack/fanout/pull/272#discussion_r4160763216
    After a 300 s run at 120k offered rows/s (telemetrygen, two spans per trace), catalog.duckdb was 762 MiB against 529 MiB of Parquet: about 474 MiB of used blocks and 288 MiB free. read_trace_parts and read_trace_candidates each held about 3.6M rows. Document that cache size scales with trace count and that free blocks are not returned without a rebuild, and consider compaction of the cache file.

  3. Trace candidates re-aggregate every overlapping part when any cached file straddles the window (low, plausible): https://github.com/labstack/fanout/pull/272#discussion_r4160763233
    Under continuous ingest the newest cached file usually straddles the window end, so most trace reads group read_trace_parts for all overlapping batches instead of reading read_trace_candidates. Restrict the parts aggregation to trace_id IN (partial) and take fully contained traces from the index, as the other branch does. Add a parity test and measure trace read cost as retention grows.

  4. The candidate index is evaluated twice in the contained-files branch (low): https://github.com/labstack/fanout/pull/272#discussion_r4160763241
    The branch added in e7dbc79a reads index once for the fully contained rows and again inside partial. On scoped reads with mixed-scope traces this repeats the read_trace_parts union. Results are unaffected. Bind it once, or share it with the rewrite in item 3.

Related: #277 (ingest CPU per row), #278 (sustained read capacity and peak RSS).

主要言語
Go
スター
6
フォーク
0
平均マージ
2時間 25分
マージ済み PR（30日）
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

labstack/fanout のほかの issue

labstack/fanout の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。