File-position checkpoints can point inside a transaction, causing --resume to fail with a missing TABLE_MAP error

Aperta
  • Qualcosa non va?
#1,776 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 10 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, mysql
Ambito
databases

Direzione di ricerca

Start in go/logic/migrator.go at Checkpoint() and resume handling, then trace currentCoordinates, LastTrxCoords, and StartSync() in go/binlog/gomysql_reader.go. Reproduce the file-position checkpoint scenario described in the issue and verify that checkpoints use a restartable transaction boundary while preserving preceding DML, allowing --resume to avoid the missing TABLE_MAP error.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Using gh-ost v1.1.11 (2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa) on Aurora MySQL 8.0 with binlog_format=ROW, without --gtid.

After running with --checkpoint --execute, stopping the process, and restarting with --resume, binlog streaming repeatedly fails with:

StreamEvents encountered unexpected error:
... invalid table id, no corresponding table map event

The streamer repeatedly reconnects at the same position. Row copying can reach 100%, but binlog processing remains stuck.

Evidence

The following is an actual checkpoint and surrounding binlog events, with database/table names anonymized and irrelevant columns omitted. Numeric positions are unchanged.

mysql> SELECT gh_ost_chk_id, gh_ost_chk_coords, gh_ost_is_cutover
       FROM app_db._table_a_ghk;

gh_ost_chk_id:     1
gh_ost_chk_coords: mysql-bin-changelog.001644:98771662
gh_ost_is_cutover: 0

mysql> SHOW BINLOG EVENTS
       IN 'mysql-bin-changelog.001644' FROM 98746801 LIMIT 10;

Pos       Event_type      End_log_pos  Info
98746801  Anonymous_Gtid   98746886     SET @@SESSION.GTID_NEXT='ANONYMOUS'
98746886  Query            98746991     BEGIN
98746991  Table_map        98747104     table_id: 5131 (app_db._other_table_gho; another concurrent migration)
98747104  Write_rows       98755290     table_id: 5131
98755290  Write_rows       98763476     table_id: 5131
98763476  Write_rows       98771662     table_id: 5131
98771662  Write_rows       98779848     table_id: 5131  <-- saved checkpoint
98779848  Write_rows       98786279     table_id: 5131 flags: STMT_END_F
98786279  Xid              98786310     COMMIT
98786310  Anonymous_Gtid   98786395     SET @@SESSION.GTID_NEXT='ANONYMOUS'

The checkpoint points to a Write_rows event after its required Table_map, rather than a transaction boundary.

Suspected cause
  • Checkpoint() saves GetCurrentBinlogCoordinates() as LastTrxCoords.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L1812-L1838

  • In file-position mode, currentCoordinates advances on every event, including intermediate row events.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L163-L176

  • Resume loads the saved coordinates and passes that position directly to StartSync(). The new reader therefore lacks the preceding table map.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L580-L595

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L92-L102

The reader separately tracks completed transactions in LastTrxCoords on XID events, but checkpoint creation does not use that value.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L216-L225

Expected behavior

File-position checkpoints should store a restartable transaction boundary, while ensuring all preceding DML has been applied, so --resume can continue.

Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
9g 19h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/gh-ost

Tutte le issue di github/gh-ost

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.