experimental_compaction: fresh reads serve frozen rows after the second compaction pass (1.6.2)

オープン
  • 何か問題がありますか？
#4,848 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
説明が足りない
活発さ
活発
技術スタック
docker, postgresql
領域
databases, distributed-systems

調査の方向性

Start by reproducing with the exact stored shape definition and the reported Postgres 15.19, file-storage, and compaction settings. Compare fresh initial reads with a persistent follower and the matching Postgres rows, and inspect the shape's log/ directory and compacted files across passes. Done means fresh reads include current values after repeated compactions, without relying on later updates to correct them.

索引モデルが issue の本文から書いたものです。

説明

Version: sync-service 1.6.2 (Docker image), Postgres 15.19, file storage on a persistent volume. Every shape is requested with
experimental_compaction=true.

Summary

Once a compaction-enabled shape has been through two compaction passes, an initial read (offset=-1, then
followed until up-to-date) returns a subset of keys at values from the end of the shape's first log chunk. The table
keeps changing, and the server keeps appending those keys' updates to the live log, but the initial read does not
return them, apart from a brief window right after a pass. Nothing is logged at any level. A client that was already following before the compaction stays
correct. A client that does a fresh initial read starts with the frozen values and corrects each key only when that
key's next update arrives.

We know #4470 is closed, and we read the comment that compaction isn't worth it compared with recreating shapes. We
are filing so the failure mode is on record for anyone still running the flag: it is silent, and it is reached by
volume within hours on a busy shape. We found no fix in the changelog up to 1.8.2.

Reproduction (one new shape, nothing else changed)

  1. Choose a table of about 12 rows where each row is updated every few seconds (one updated_at per row). Request a
    new shape on it with a where clause, experimental_compaction=true, offset=-1. The log grows about
    150 KB/min.
  2. Every 60 s, take three readings: (a) a fresh initial read (offset=-1, followed to up-to-date); (b) a
    follower, a single client that keeps its handle and offset; (c) the table in Postgres with the same where.
    Compare updated_at per key. A key is stale if the server's value is more than 60 s behind the table's.
  3. Watch the shape's log/ directory.

What we observed

time after creation log fresh read follower
0-230 min chunk index grows by 64 B at each ~11.1 MB boundary (3 boundaries) current (225 of 225 readings) current
+230 min first compaction: one chunk folded into log.compacted.<a> (jsonfile 11,992 B, keyfile 774 B) still current (8 of 8 readings); updates without an lsn now appear in the initial read current
+238 min second compaction: log.compacted.<b> is byte-identical to <a> (same sha256 for jsonfile and keyfile) stale from the next reading, every reading through +255 min (18 of 18): 6-9 of 12 keys frozen at the timestamp of the end of the first chunk, with the gap growing by the time between readings current
+255 min a new follower starts from a fresh read starts with 6 of 12 keys stale, and all are corrected within 60 s as each key's next update arrives —
+314 min third compaction: log.compacted.<c> byte-identical to <a> and <b> still stale (7 of 12, worst ~4 h) current
+320 min fourth compaction, 5.5 min later: byte-identical again current for one reading, 25 s after the pass; stale again from the next reading (same frozen values) with no further pass current

Server log: one [info] Creating new shape … with handle … line for the shape, and nothing else for it. No warning or
error, and no line mentioning compaction.

Long-running shapes show the same thing. On a second deployment of the same stack (same version and flag), 12 of
14 compaction-enabled shapes were stale when read the same way, the worst by 3.2 days. In one shape, every compacted
file held the same single row (5,693 B) from three days earlier, while that row was being rewritten about twice a
second. Each pass carried it forward unchanged. Two shapes that read current at one time read older values on a
re-read four hours later.

Notes that may help

  • What triggers compaction appeared to be volume. The first pass came 9 minutes after the third complete chunk,
    and not after the second. A low-volume shape with one complete chunk never compacted in 6 days.
  • Every pass rewrote identical bytes, so nothing after the first chunk is ever folded in. The later updates are still
    in the live log, because a follower receives them. So the initial read appears to skip part of the live log after
    the compacted portion. We have not read the code to confirm this.
  • Replaying a definition creates a shape. While investigating, a replay without the stored explicit columns
    created a new shape instead of reading the existing one. A reproduction needs the exact stored definition.

The workaround we are moving to

Compaction off (shapes created without the flag; existing shapes need their storage cleared, since the flag is fixed
per shape), and shapes recreated on a timer to bound log length.

主要言語
TypeScript
スター
10.4k
フォーク
399
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

electric-sql/electric のほかの issue

electric-sql/electric の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。