experimental_compaction: fresh reads serve frozen rows after the second compaction pass (1.6.2)

Đang mở
  • Có gì không ổn?
#4,848 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Cần làm rõ
Mức độ hoạt động
Sôi nổi
Công nghệ
docker, postgresql
Lĩnh vực
databases, distributed-systems

Hướng nghiên cứu

Start by reproducing with the exact stored shape definition and the reported Postgres 15.19, file-storage, and compaction settings. Compare fresh initial reads with a persistent follower and the matching Postgres rows, and inspect the shape's log/ directory and compacted files across passes. Done means fresh reads include current values after repeated compactions, without relying on later updates to correct them.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Version: sync-service 1.6.2 (Docker image), Postgres 15.19, file storage on a persistent volume. Every shape is requested with
experimental_compaction=true.

Summary

Once a compaction-enabled shape has been through two compaction passes, an initial read (offset=-1, then
followed until up-to-date) returns a subset of keys at values from the end of the shape's first log chunk. The table
keeps changing, and the server keeps appending those keys' updates to the live log, but the initial read does not
return them, apart from a brief window right after a pass. Nothing is logged at any level. A client that was already following before the compaction stays
correct. A client that does a fresh initial read starts with the frozen values and corrects each key only when that
key's next update arrives.

We know #4470 is closed, and we read the comment that compaction isn't worth it compared with recreating shapes. We
are filing so the failure mode is on record for anyone still running the flag: it is silent, and it is reached by
volume within hours on a busy shape. We found no fix in the changelog up to 1.8.2.

Reproduction (one new shape, nothing else changed)

  1. Choose a table of about 12 rows where each row is updated every few seconds (one updated_at per row). Request a
    new shape on it with a where clause, experimental_compaction=true, offset=-1. The log grows about
    150 KB/min.
  2. Every 60 s, take three readings: (a) a fresh initial read (offset=-1, followed to up-to-date); (b) a
    follower, a single client that keeps its handle and offset; (c) the table in Postgres with the same where.
    Compare updated_at per key. A key is stale if the server's value is more than 60 s behind the table's.
  3. Watch the shape's log/ directory.

What we observed

time after creation log fresh read follower
0-230 min chunk index grows by 64 B at each ~11.1 MB boundary (3 boundaries) current (225 of 225 readings) current
+230 min first compaction: one chunk folded into log.compacted.<a> (jsonfile 11,992 B, keyfile 774 B) still current (8 of 8 readings); updates without an lsn now appear in the initial read current
+238 min second compaction: log.compacted.<b> is byte-identical to <a> (same sha256 for jsonfile and keyfile) stale from the next reading, every reading through +255 min (18 of 18): 6-9 of 12 keys frozen at the timestamp of the end of the first chunk, with the gap growing by the time between readings current
+255 min a new follower starts from a fresh read starts with 6 of 12 keys stale, and all are corrected within 60 s as each key's next update arrives —
+314 min third compaction: log.compacted.<c> byte-identical to <a> and <b> still stale (7 of 12, worst ~4 h) current
+320 min fourth compaction, 5.5 min later: byte-identical again current for one reading, 25 s after the pass; stale again from the next reading (same frozen values) with no further pass current

Server log: one [info] Creating new shape … with handle … line for the shape, and nothing else for it. No warning or
error, and no line mentioning compaction.

Long-running shapes show the same thing. On a second deployment of the same stack (same version and flag), 12 of
14 compaction-enabled shapes were stale when read the same way, the worst by 3.2 days. In one shape, every compacted
file held the same single row (5,693 B) from three days earlier, while that row was being rewritten about twice a
second. Each pass carried it forward unchanged. Two shapes that read current at one time read older values on a
re-read four hours later.

Notes that may help

  • What triggers compaction appeared to be volume. The first pass came 9 minutes after the third complete chunk,
    and not after the second. A low-volume shape with one complete chunk never compacted in 6 days.
  • Every pass rewrote identical bytes, so nothing after the first chunk is ever folded in. The later updates are still
    in the live log, because a follower receives them. So the initial read appears to skip part of the live log after
    the compacted portion. We have not read the code to confirm this.
  • Replaying a definition creates a shape. While investigating, a replay without the stored explicit columns
    created a new shape instead of reading the existing one. A reproduction needs the exact stored definition.

The workaround we are moving to

Compaction off (shapes created without the flag; existing shapes need their storage cleared, since the flag is fixed
per shape), and shapes recreated on a timer to bound log length.

Ngôn ngữ chính
TypeScript
Star
10.4k
Fork
399
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của electric-sql/electric

Tất cả issue của electric-sql/electric

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.