Parquet partition unreadable after an unclean restart under ADAPTIVE commit mode

Aperta
  • Qualcosa non va?
#8,271 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
java
Ambito
database, databases

Direzione di ricerca

Read core/src/main/java/io/questdb/cairo/O3PartitionJob.java around the reported file-size publication, then trace how TableWriter.o3ConsumePartitionUpdateSink records that size in _txn. Run the two crash-sweep methods in probes/M1StaleParquetTailProbeTest.java with -Dquestdb.fuzz.nightly=true. Done means recovery after each simulated crash leaves the table unsuspended and its Parquet partition readable.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug storage

Status: PRE-EXISTING (same result at the PR base fe6b0412d1 and at the PR #7684 head 4e5a6e3a09).

Summary

Under cairo.commit.mode=adaptive, two in-place O3 updates of one Parquet partition between two durable epochs,
followed by a power loss or a process kill (kill -9, OOM killer, JVM crash) before the next epoch, leave the partition
unreadable after restart: every query that touches it fails with
invalid _pm file: failed to resolve footer, or WAL apply suspends the table with
_pm tail does not match current parquet file size. RESUME WAL does not help, because _txn already names a
Parquet file size that no _pm footer describes.

Trigger

Settings: cairo.commit.mode=adaptive, a WAL table, a partition converted to Parquet with more than one row group
(and dead-bytes thresholds that keep the updates in place, e.g. the defaults with small O3 batches, or
cairo.partition.encoder.parquet.o3.rewrite.unused.ratio=1.0 in tests).

  1. CREATE TABLE t (id LONG, ts TIMESTAMP, v LONG) TIMESTAMP(ts) PARTITION BY DAY WAL;
    12 rows at 2h spacing on 2024-01-01, 2 rows on 2024-01-02, row group size 4;
    ALTER TABLE t CONVERT PARTITION TO PARQUET LIST '2024-01-01';
  2. Insert a row at 2024-01-01T03:00 (merges into row group 0 in place), then a row at 2024-01-01T17:00
    (merges into row group 2 in place), with no durable epoch between them.
  3. Power loss right after the second update's data.parquet fsync, or a process kill at any point after the second
    update wrote its bytes and before the next epoch.
  4. Restart. Recovery restores _txn/_cv from the save point taken before step 2 and replays the WAL.

Reproducer (crash-sweep tests, run with -Dquestdb.fuzz.nightly=true):
probes/M1StaleParquetTailProbeTest.java

  • testTwoInPlaceO3UpdatesBetweenSavePointsCrashSafeW0 (power loss): N=16, fails at crash point 16.
  • testTwoInPlaceO3UpdatesBetweenSavePointsKillSafe (process kill): N=54, fails at kill point 40.

Result

At HEAD and at the PR base, identically:

io.questdb.cairo.CairoException: [0] invalid _pm file: failed to resolve footer [path=.../cf_tail~18/2024-01-01.1/_pm]
    at io.questdb.cairo.TableReader.openParquetMetadata(TableReader.java:1622)

When the replayed transactions are applied one at a time (a replace-range commit is never batched), the second
replay fails instead and the table is suspended:

_pm tail does not match current parquet file size [path=.../2024-01-01.1/_pm, parquetFileSize=7798]

Expected: the table recovers to the last committed state, unsuspended, with the partition readable.

Root cause

O3PartitionJob.processParquetPartition0 publishes the physical length of data.parquet as the committed Parquet
file size:

  • HEAD: core/src/main/java/io/questdb/cairo/O3PartitionJob.java:3071
    (final long fileSize = Files.length(path.$());, written to sink slot 7 at :3079)
  • PR base: the same statement at O3PartitionJob.java:2829

TableWriter.o3ConsumePartitionUpdateSink stores that value in _txn (setPartitionParquetFileSizeByRawIndex).

An in-place update writes from the committed size (writeFileSize = parquetSize) and never truncates the file.
After the unclean exit, data.parquet still holds the second update's bytes (made durable by its data fsync, or kept
in the page cache by a kill), while _txn is rewound to the save point. The replay of the first update rewrites a
prefix of that tail; its logical size (newParquetSize from partitionUpdater.updateFileMetadata(), e.g. 5806 bytes)
is smaller than the physical length (7798 bytes), and the physical length is what _txn gets. No _pm footer
describes 7798, so the reader cannot resolve one and the next update cannot either.

Suggested fix

Publish the size the updater produced (newParquetSize) in sink slot 7 instead of Files.length(path), in both
update and rewrite mode; the bytes past it are a dead tail that readers ignore and the next update overwrites, as the
rollback path already assumes. Alternatively truncate data.parquet to newParquetSize before its fsync. The rewrite
mode is probably exposed too: a txn-named directory left by an interrupted attempt at the same txn keeps its
data.parquet, which openRW does not truncate, and a replay that batches transactions differently writes a
shorter file into it (not reproduced).

Regression test: the two crash-sweep methods of the reproducer, which pass once the published size is the logical
one, plus a sweep that keeps both in-place updates in one save-point window over an indexed SYMBOL (the PR #7684 M1
sweep takes an epoch between its two in-place commits to stay clear of this bug).

Related

#7921 shows the same symptom (_txn names a Parquet size no _pm footer describes) through a different trigger (cairo.repeat.migration.from.version).

Lingua principale
Java
Stelle
17.5k
Fork
1.7k
Merge medio
5g 11h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di questdb/questdb

Tutte le issue di questdb/questdb

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.