experimental_compaction: fresh reads serve frozen rows after the second compaction pass (1.6.2)

未关闭
  • 哪里不对？
#4,848 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
活跃
技术栈
docker, postgresql
领域
databases, distributed-systems

调研方向

Start by reproducing with the exact stored shape definition and the reported Postgres 15.19, file-storage, and compaction settings. Compare fresh initial reads with a persistent follower and the matching Postgres rows, and inspect the shape's log/ directory and compacted files across passes. Done means fresh reads include current values after repeated compactions, without relying on later updates to correct them.

由索引模型根据 Issue 内容生成。

描述

Version: sync-service 1.6.2 (Docker image), Postgres 15.19, file storage on a persistent volume. Every shape is requested with
experimental_compaction=true.

Summary

Once a compaction-enabled shape has been through two compaction passes, an initial read (offset=-1, then
followed until up-to-date) returns a subset of keys at values from the end of the shape's first log chunk. The table
keeps changing, and the server keeps appending those keys' updates to the live log, but the initial read does not
return them, apart from a brief window right after a pass. Nothing is logged at any level. A client that was already following before the compaction stays
correct. A client that does a fresh initial read starts with the frozen values and corrects each key only when that
key's next update arrives.

We know #4470 is closed, and we read the comment that compaction isn't worth it compared with recreating shapes. We
are filing so the failure mode is on record for anyone still running the flag: it is silent, and it is reached by
volume within hours on a busy shape. We found no fix in the changelog up to 1.8.2.

Reproduction (one new shape, nothing else changed)

  1. Choose a table of about 12 rows where each row is updated every few seconds (one updated_at per row). Request a
    new shape on it with a where clause, experimental_compaction=true, offset=-1. The log grows about
    150 KB/min.
  2. Every 60 s, take three readings: (a) a fresh initial read (offset=-1, followed to up-to-date); (b) a
    follower, a single client that keeps its handle and offset; (c) the table in Postgres with the same where.
    Compare updated_at per key. A key is stale if the server's value is more than 60 s behind the table's.
  3. Watch the shape's log/ directory.

What we observed

time after creation log fresh read follower
0-230 min chunk index grows by 64 B at each ~11.1 MB boundary (3 boundaries) current (225 of 225 readings) current
+230 min first compaction: one chunk folded into log.compacted.<a> (jsonfile 11,992 B, keyfile 774 B) still current (8 of 8 readings); updates without an lsn now appear in the initial read current
+238 min second compaction: log.compacted.<b> is byte-identical to <a> (same sha256 for jsonfile and keyfile) stale from the next reading, every reading through +255 min (18 of 18): 6-9 of 12 keys frozen at the timestamp of the end of the first chunk, with the gap growing by the time between readings current
+255 min a new follower starts from a fresh read starts with 6 of 12 keys stale, and all are corrected within 60 s as each key's next update arrives —
+314 min third compaction: log.compacted.<c> byte-identical to <a> and <b> still stale (7 of 12, worst ~4 h) current
+320 min fourth compaction, 5.5 min later: byte-identical again current for one reading, 25 s after the pass; stale again from the next reading (same frozen values) with no further pass current

Server log: one [info] Creating new shape … with handle … line for the shape, and nothing else for it. No warning or
error, and no line mentioning compaction.

Long-running shapes show the same thing. On a second deployment of the same stack (same version and flag), 12 of
14 compaction-enabled shapes were stale when read the same way, the worst by 3.2 days. In one shape, every compacted
file held the same single row (5,693 B) from three days earlier, while that row was being rewritten about twice a
second. Each pass carried it forward unchanged. Two shapes that read current at one time read older values on a
re-read four hours later.

Notes that may help

  • What triggers compaction appeared to be volume. The first pass came 9 minutes after the third complete chunk,
    and not after the second. A low-volume shape with one complete chunk never compacted in 6 days.
  • Every pass rewrote identical bytes, so nothing after the first chunk is ever folded in. The later updates are still
    in the live log, because a follower receives them. So the initial read appears to skip part of the live log after
    the compacted portion. We have not read the code to confirm this.
  • Replaying a definition creates a shape. While investigating, a replay without the stored explicit columns
    created a new shape instead of reading the existing one. A reproduction needs the exact stored definition.

The workaround we are moving to

Compaction off (shapes created without the flag; existing shapes need their storage cleared, since the flag is fixed
per shape), and shapes recreated on a timer to bound log length.

主要语言
TypeScript
星标
10.4k
派生
399
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

electric-sql/electric 的其他 Issue

查看 electric-sql/electric 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。