--resume with --gtid can skip the rest of a large transaction (checkpoint written mid-transaction): silent data loss

Aperta
  • Qualcosa non va?
#1,781 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 10 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, mysql
Ambito
databases

Direzione di ricerca

Read go/binlog/gomysql_reader.go around GTIDEvent and go/logic/migrator.go around Migrator.Checkpoint() and onApplyEventStruct. Trace when coordinates advance during a multi-batch transaction, then reproduce with --gtid --checkpoint and a large transaction. Done means a checkpoint and --resume cannot skip unapplied rows when a crash occurs mid-transaction.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

With --gtid --checkpoint, a checkpoint can be written while a multi-row transaction is only partly applied to the
ghost table. --resume then starts the binlog stream from a GTID set that already contains that transaction, so the
server never sends it again and the rest of its rows are silently lost. The migration completes normally (exit code
0, Done migrating) and the cut-over puts the stale ghost table live.

How it happens (v1.1.11)
  • GoMySQLReader advances its coordinates on the GTIDEvent, i.e. before the transaction's row events
    (go/binlog/gomysql_reader.go, case *replication.GTIDEvent): the running GTID set already includes the
    transaction being streamed.
  • Migrator.Checkpoint() takes eventsStreamer.GetCurrentBinlogCoordinates() as LastTrxCoords and writes the
    checkpoint as soon as coords.SmallerThanOrEquals(applier.CurrentCoordinates) (go/logic/migrator.go, around
    line 1815).
  • The applier sets CurrentCoordinates = eventStruct.coords after every DML batch (onApplyEventStruct, around line
    1797) — after the first batch of a large transaction its coordinates already equal the streamer's.
  • So a checkpoint taken during a large transaction records its GTID after only --dml-batch-size rows were applied.
    On --resume, InitialStreamerCoords = lastCheckpoint.LastTrxCoords and the replica stream starts with that set:
    the transaction is considered executed and is skipped entirely.
Reproduction

MySQL 9.7 (Percona Server 9.7.1-1; nothing version specific is involved), single node, GTID on, a table t of
250,000 rows (id PK, chat_id, v), gh-ost v1.1.11 built from the tag:

gh-ost ... --database=vt --table=t --alter="ADD INDEX chat_v (chat_id, v)" \
  --allow-on-master --assume-rbr --gtid --checkpoint --checkpoint-seconds=10 --dml-batch-size=10 \
  --cut-over=atomic --postpone-cut-over-flag-file=/tmp/postpone --execute
  1. Let the row copy complete (cut-over postponed).
  2. Run one big transaction: UPDATE vt.t SET v = v + 1000; (271,299 rows, GTID …:999999).
  3. Wait until the newest row of _t_ghk has gh_ost_chk_coords containing that GTID while _t_gho has only part of
    the update (here: checkpoint …:1-53495:999999 written when 14,360 of 271,299 rows were updated in _t_gho).
  4. kill -9 gh-ost.
  5. Restart the same command with --resume:
    Resuming from checkpoint coords=…:1-53495:999999 range_min=262190 range_max=262190 iteration=249.
  6. Compare: 256,889 rows of _t_gho still have the old v while t has the new one; no rows are missing.
  7. Remove the postpone flag: cut-over succeeds, exit code 0 — the live table now has 256,889 rows with the update
    lost (the old table _t_<ts>_del has them right).

Before the kill, during the copy with concurrent inserts/updates/deletes, t and _t_gho matched by checksum, so
the loss comes from the resume only.

Impact

Silent data loss on resume after any crash or kill that happens while a large transaction is being applied: bulk
UPDATE/DELETE, backfills, archival jobs. A lost DELETE brings deleted rows back.

Possible fix

Checkpoint only on a transaction boundary: record the coordinates of the last transaction applied completely (e.g.
advance the applier's checkpointable coordinates on the transaction's XID/commit event rather than per batch), or
make Checkpoint() use the coordinates of the previous transaction while one is in progress. Until then, a note in
doc/resume.md that --resume is unsafe after a crash in the middle of a large transaction would help.

Related: #1780 (a different cut-over issue found while testing the same setup).

Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
9g 19h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/gh-ost

Tutte le issue di github/gh-ost

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.