experimental_compaction: fresh reads serve frozen rows after the second compaction pass (1.6.2)

Abierto
  • ¿Algo no está bien?
#4,848 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
docker, postgresql
Área
databases, distributed-systems

Línea de trabajo

Start by reproducing with the exact stored shape definition and the reported Postgres 15.19, file-storage, and compaction settings. Compare fresh initial reads with a persistent follower and the matching Postgres rows, and inspect the shape's log/ directory and compacted files across passes. Done means fresh reads include current values after repeated compactions, without relying on later updates to correct them.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Version: sync-service 1.6.2 (Docker image), Postgres 15.19, file storage on a persistent volume. Every shape is requested with
experimental_compaction=true.

Summary

Once a compaction-enabled shape has been through two compaction passes, an initial read (offset=-1, then
followed until up-to-date) returns a subset of keys at values from the end of the shape's first log chunk. The table
keeps changing, and the server keeps appending those keys' updates to the live log, but the initial read does not
return them, apart from a brief window right after a pass. Nothing is logged at any level. A client that was already following before the compaction stays
correct. A client that does a fresh initial read starts with the frozen values and corrects each key only when that
key's next update arrives.

We know #4470 is closed, and we read the comment that compaction isn't worth it compared with recreating shapes. We
are filing so the failure mode is on record for anyone still running the flag: it is silent, and it is reached by
volume within hours on a busy shape. We found no fix in the changelog up to 1.8.2.

Reproduction (one new shape, nothing else changed)

  1. Choose a table of about 12 rows where each row is updated every few seconds (one updated_at per row). Request a
    new shape on it with a where clause, experimental_compaction=true, offset=-1. The log grows about
    150 KB/min.
  2. Every 60 s, take three readings: (a) a fresh initial read (offset=-1, followed to up-to-date); (b) a
    follower, a single client that keeps its handle and offset; (c) the table in Postgres with the same where.
    Compare updated_at per key. A key is stale if the server's value is more than 60 s behind the table's.
  3. Watch the shape's log/ directory.

What we observed

time after creation log fresh read follower
0-230 min chunk index grows by 64 B at each ~11.1 MB boundary (3 boundaries) current (225 of 225 readings) current
+230 min first compaction: one chunk folded into log.compacted.<a> (jsonfile 11,992 B, keyfile 774 B) still current (8 of 8 readings); updates without an lsn now appear in the initial read current
+238 min second compaction: log.compacted.<b> is byte-identical to <a> (same sha256 for jsonfile and keyfile) stale from the next reading, every reading through +255 min (18 of 18): 6-9 of 12 keys frozen at the timestamp of the end of the first chunk, with the gap growing by the time between readings current
+255 min a new follower starts from a fresh read starts with 6 of 12 keys stale, and all are corrected within 60 s as each key's next update arrives —
+314 min third compaction: log.compacted.<c> byte-identical to <a> and <b> still stale (7 of 12, worst ~4 h) current
+320 min fourth compaction, 5.5 min later: byte-identical again current for one reading, 25 s after the pass; stale again from the next reading (same frozen values) with no further pass current

Server log: one [info] Creating new shape … with handle … line for the shape, and nothing else for it. No warning or
error, and no line mentioning compaction.

Long-running shapes show the same thing. On a second deployment of the same stack (same version and flag), 12 of
14 compaction-enabled shapes were stale when read the same way, the worst by 3.2 days. In one shape, every compacted
file held the same single row (5,693 B) from three days earlier, while that row was being rewritten about twice a
second. Each pass carried it forward unchanged. Two shapes that read current at one time read older values on a
re-read four hours later.

Notes that may help

  • What triggers compaction appeared to be volume. The first pass came 9 minutes after the third complete chunk,
    and not after the second. A low-volume shape with one complete chunk never compacted in 6 days.
  • Every pass rewrote identical bytes, so nothing after the first chunk is ever folded in. The later updates are still
    in the live log, because a follower receives them. So the initial read appears to skip part of the live log after
    the compacted portion. We have not read the code to confirm this.
  • Replaying a definition creates a shape. While investigating, a replay without the stored explicit columns
    created a new shape instead of reading the existing one. A reproduction needs the exact stored definition.

The workaround we are moving to

Compaction off (shapes created without the flag; existing shapes need their storage cleared, since the flag is fixed
per shape), and shapes recreated on a timer to bound log length.

Lenguaje dominante
TypeScript
Estrellas
10.4k
Forks
399
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de electric-sql/electric

Todos los issues de electric-sql/electric

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.