--resume with --gtid can skip the rest of a large transaction (checkpoint written mid-transaction): silent data loss

Đang mở
  • Có gì không ổn?
#1,781 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 10 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go, mysql
Lĩnh vực
databases

Hướng nghiên cứu

Read go/binlog/gomysql_reader.go around GTIDEvent and go/logic/migrator.go around Migrator.Checkpoint() and onApplyEventStruct. Trace when coordinates advance during a multi-batch transaction, then reproduce with --gtid --checkpoint and a large transaction. Done means a checkpoint and --resume cannot skip unapplied rows when a crash occurs mid-transaction.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

With --gtid --checkpoint, a checkpoint can be written while a multi-row transaction is only partly applied to the
ghost table. --resume then starts the binlog stream from a GTID set that already contains that transaction, so the
server never sends it again and the rest of its rows are silently lost. The migration completes normally (exit code
0, Done migrating) and the cut-over puts the stale ghost table live.

How it happens (v1.1.11)
  • GoMySQLReader advances its coordinates on the GTIDEvent, i.e. before the transaction's row events
    (go/binlog/gomysql_reader.go, case *replication.GTIDEvent): the running GTID set already includes the
    transaction being streamed.
  • Migrator.Checkpoint() takes eventsStreamer.GetCurrentBinlogCoordinates() as LastTrxCoords and writes the
    checkpoint as soon as coords.SmallerThanOrEquals(applier.CurrentCoordinates) (go/logic/migrator.go, around
    line 1815).
  • The applier sets CurrentCoordinates = eventStruct.coords after every DML batch (onApplyEventStruct, around line
    1797) — after the first batch of a large transaction its coordinates already equal the streamer's.
  • So a checkpoint taken during a large transaction records its GTID after only --dml-batch-size rows were applied.
    On --resume, InitialStreamerCoords = lastCheckpoint.LastTrxCoords and the replica stream starts with that set:
    the transaction is considered executed and is skipped entirely.
Reproduction

MySQL 9.7 (Percona Server 9.7.1-1; nothing version specific is involved), single node, GTID on, a table t of
250,000 rows (id PK, chat_id, v), gh-ost v1.1.11 built from the tag:

gh-ost ... --database=vt --table=t --alter="ADD INDEX chat_v (chat_id, v)" \
  --allow-on-master --assume-rbr --gtid --checkpoint --checkpoint-seconds=10 --dml-batch-size=10 \
  --cut-over=atomic --postpone-cut-over-flag-file=/tmp/postpone --execute
  1. Let the row copy complete (cut-over postponed).
  2. Run one big transaction: UPDATE vt.t SET v = v + 1000; (271,299 rows, GTID …:999999).
  3. Wait until the newest row of _t_ghk has gh_ost_chk_coords containing that GTID while _t_gho has only part of
    the update (here: checkpoint …:1-53495:999999 written when 14,360 of 271,299 rows were updated in _t_gho).
  4. kill -9 gh-ost.
  5. Restart the same command with --resume:
    Resuming from checkpoint coords=…:1-53495:999999 range_min=262190 range_max=262190 iteration=249.
  6. Compare: 256,889 rows of _t_gho still have the old v while t has the new one; no rows are missing.
  7. Remove the postpone flag: cut-over succeeds, exit code 0 — the live table now has 256,889 rows with the update
    lost (the old table _t_<ts>_del has them right).

Before the kill, during the copy with concurrent inserts/updates/deletes, t and _t_gho matched by checksum, so
the loss comes from the resume only.

Impact

Silent data loss on resume after any crash or kill that happens while a large transaction is being applied: bulk
UPDATE/DELETE, backfills, archival jobs. A lost DELETE brings deleted rows back.

Possible fix

Checkpoint only on a transaction boundary: record the coordinates of the last transaction applied completely (e.g.
advance the applier's checkpointable coordinates on the transaction's XID/commit event rather than per batch), or
make Checkpoint() use the coordinates of the previous transaction while one is in progress. Until then, a note in
doc/resume.md that --resume is unsafe after a crash in the middle of a large transaction would help.

Related: #1780 (a different cut-over issue found while testing the same setup).

Ngôn ngữ chính
Go
Star
13.6k
Fork
1.4k
Merge trung bình
9 ngày 19 giờ
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của github/gh-ost

Tất cả issue của github/gh-ost

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.