experimental_compaction: fresh reads serve frozen rows after the second compaction pass (1.6.2)

Aperta
  • Qualcosa non va?
#4,848 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
docker, postgresql
Ambito
databases, distributed-systems

Direzione di ricerca

Start by reproducing with the exact stored shape definition and the reported Postgres 15.19, file-storage, and compaction settings. Compare fresh initial reads with a persistent follower and the matching Postgres rows, and inspect the shape's log/ directory and compacted files across passes. Done means fresh reads include current values after repeated compactions, without relying on later updates to correct them.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Version: sync-service 1.6.2 (Docker image), Postgres 15.19, file storage on a persistent volume. Every shape is requested with
experimental_compaction=true.

Summary

Once a compaction-enabled shape has been through two compaction passes, an initial read (offset=-1, then
followed until up-to-date) returns a subset of keys at values from the end of the shape's first log chunk. The table
keeps changing, and the server keeps appending those keys' updates to the live log, but the initial read does not
return them, apart from a brief window right after a pass. Nothing is logged at any level. A client that was already following before the compaction stays
correct. A client that does a fresh initial read starts with the frozen values and corrects each key only when that
key's next update arrives.

We know #4470 is closed, and we read the comment that compaction isn't worth it compared with recreating shapes. We
are filing so the failure mode is on record for anyone still running the flag: it is silent, and it is reached by
volume within hours on a busy shape. We found no fix in the changelog up to 1.8.2.

Reproduction (one new shape, nothing else changed)

  1. Choose a table of about 12 rows where each row is updated every few seconds (one updated_at per row). Request a
    new shape on it with a where clause, experimental_compaction=true, offset=-1. The log grows about
    150 KB/min.
  2. Every 60 s, take three readings: (a) a fresh initial read (offset=-1, followed to up-to-date); (b) a
    follower, a single client that keeps its handle and offset; (c) the table in Postgres with the same where.
    Compare updated_at per key. A key is stale if the server's value is more than 60 s behind the table's.
  3. Watch the shape's log/ directory.

What we observed

time after creation log fresh read follower
0-230 min chunk index grows by 64 B at each ~11.1 MB boundary (3 boundaries) current (225 of 225 readings) current
+230 min first compaction: one chunk folded into log.compacted.<a> (jsonfile 11,992 B, keyfile 774 B) still current (8 of 8 readings); updates without an lsn now appear in the initial read current
+238 min second compaction: log.compacted.<b> is byte-identical to <a> (same sha256 for jsonfile and keyfile) stale from the next reading, every reading through +255 min (18 of 18): 6-9 of 12 keys frozen at the timestamp of the end of the first chunk, with the gap growing by the time between readings current
+255 min a new follower starts from a fresh read starts with 6 of 12 keys stale, and all are corrected within 60 s as each key's next update arrives —
+314 min third compaction: log.compacted.<c> byte-identical to <a> and <b> still stale (7 of 12, worst ~4 h) current
+320 min fourth compaction, 5.5 min later: byte-identical again current for one reading, 25 s after the pass; stale again from the next reading (same frozen values) with no further pass current

Server log: one [info] Creating new shape … with handle … line for the shape, and nothing else for it. No warning or
error, and no line mentioning compaction.

Long-running shapes show the same thing. On a second deployment of the same stack (same version and flag), 12 of
14 compaction-enabled shapes were stale when read the same way, the worst by 3.2 days. In one shape, every compacted
file held the same single row (5,693 B) from three days earlier, while that row was being rewritten about twice a
second. Each pass carried it forward unchanged. Two shapes that read current at one time read older values on a
re-read four hours later.

Notes that may help

  • What triggers compaction appeared to be volume. The first pass came 9 minutes after the third complete chunk,
    and not after the second. A low-volume shape with one complete chunk never compacted in 6 days.
  • Every pass rewrote identical bytes, so nothing after the first chunk is ever folded in. The later updates are still
    in the live log, because a follower receives them. So the initial read appears to skip part of the live log after
    the compacted portion. We have not read the code to confirm this.
  • Replaying a definition creates a shape. While investigating, a replay without the stored explicit columns
    created a new shape instead of reading the existing one. A reproduction needs the exact stored definition.

The workaround we are moving to

Compaction off (shapes created without the flag; existing shapes need their storage cleared, since the flag is fixed
per shape), and shapes recreated on a timer to bound log length.

Lingua principale
TypeScript
Stelle
10.4k
Fork
399
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di electric-sql/electric

Tutte le issue di electric-sql/electric

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.