SeqnosByKey_Bucketed writes an empty file per key for every bucket since seqno 0

Đang mở
  • Có gì không ổn?
#8,673 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp
Lĩnh vực
databases, performance

Hướng nghiên cứu

Start with src/indexing/strategies/seqnos_by_key_bucketed.cpp, especially visit_entry, store_empty_buckets, query_bucket, and current_results; the issue links directly to these sections. Trace how missing buckets are distinguished from empty ones and how queries use current_results. Done means sparse keys no longer generate or fetch empty buckets, queries still detect lost data correctly, and per-key memory use is bounded; the issue presents design ideas but leaves the approach open.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Describe the bug

SeqnosByKey_Bucketed stores a file per key for every bucket from seqno 0 up to the key's current bucket, including empty ones:

  • The first time it sees a key at seqno S, it stores an empty bucket for every bucket before S (visit_entry, store_empty_buckets). With 1,000 seqnos per bucket, a key first written at seqno 5,000,000 costs 5,000 files.
  • When a key is written again after a gap, it stores an empty bucket for every bucket in the gap (L384-L388).

So the index costs about keys × seqno / bucket size files, however few writes each key has. The empty files are needed today because a query treats a missing bucket as lost data and re-indexes from seqno 0 (L254-L283).

Separately, current_results keeps an entry for every key ever indexed and never frees it (L24-L26). Queries depend on that, since a key without an entry is treated as never written (L311-L322), so entries can't just be evicted.

Each node builds all of this on its own, and again after every restart, because .index is wiped on start (enclave_lfs_access.h#L295-L302).

We use two LazyStrategy<SeqnosByKey_Bucketed<...>> indexes (1,000 seqnos per bucket, 200 buckets) keyed by collection and by (collection, tag) and we see the following:

  • In a 3-node sandbox on 7.0.23, building the index at keys × seqno ≈ 1.6e9 wrote 789k files: 3.3 GB on disk (about 4.2 KB each). Node RSS went up about 323 MiB and didn't come back after the build. We didn't dig into the RSS part. The LFS keeps no per-file state, so it may be the queued async writes plus allocator retention.
  • On one production instance, CCF logged 182k Writing LFS file operations slower than 10 ms over three weeks, up to 4.3 s each. All of them were 108-116 byte files, so empty or single-entry buckets.
  • Range queries on sparse keys also fetch every empty bucket in the range, which churns the shared bucket LRU.

To Reproduce

A standalone harness that calls visit_entry and get_write_txs_impl on the real strategy (main at 00d0976e7, 1,000 seqnos per bucket, 200 buckets), with an in-memory LFS that counts store() calls:

  • Key A first written at seqno 1: no files.
  • Key B first written at seqno 1,000,000: 1,000 files, all empty.
  • Key A written again at seqno 2,000,000: 2,000 files (1 with data, 1,999 empty).
  • 100 new keys first written around seqno 5,000,000: 500,000 empty files.
  • A query for B over the 198,000 seqnos before its first write: 198 bucket fetches, all empty.
  • 200,000 distinct keys: 200,000 current_results entries (about 160 B each with 21-byte keys), still there after later writes move the watermark on.
  • With an LFS that drops empty buckets, the next query on a range in the gap resets the watermark to 0 and clears every key.

Happy to share the harness used for checking this.

Expected behavior

Index cost should grow with the number of writes, not with keys × ledger length, and per-key memory should be bounded. A couple of ideas, but you know this code better:

  • Don't store empty buckets. Have each stored bucket, and the in-memory current one, record the start of the key's previous non-empty bucket, so a query can tell an empty bucket from a lost file and only fetches buckets with data.
  • Flush and drop current_results entries for keys that haven't been written for a while, keeping a small marker on disk so queries still know the key exists.

Environment information

CCF 7.0.23. The code is unchanged on main at 00d0976e71afc0613328225e6f4074a78f52a592.

Additional context

An app can't bound this index by key count alone, since a key created late in a ledger's life costs seqno / bucket size files. The workaround we're looking at is turning the index off above a keys × seqno threshold, but that makes range queries on exactly those ledgers about 2x slower.

Ngôn ngữ chính
C++
Star
876
Fork
264
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
127

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/CCF

Tất cả issue của microsoft/CCF

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.