File-position checkpoints can point inside a transaction, causing --resume to fail with a missing TABLE_MAP error

Offen
  • Stimmt etwas nicht?
#1,776 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 10 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
68/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
go, mysql
Bereich
databases

Rechercherichtung

Start in go/logic/migrator.go at Checkpoint() and resume handling, then trace currentCoordinates, LastTrxCoords, and StartSync() in go/binlog/gomysql_reader.go. Reproduce the file-position checkpoint scenario described in the issue and verify that checkpoints use a restartable transaction boundary while preserving preceding DML, allowing --resume to avoid the missing TABLE_MAP error.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Using gh-ost v1.1.11 (2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa) on Aurora MySQL 8.0 with binlog_format=ROW, without --gtid.

After running with --checkpoint --execute, stopping the process, and restarting with --resume, binlog streaming repeatedly fails with:

StreamEvents encountered unexpected error:
... invalid table id, no corresponding table map event

The streamer repeatedly reconnects at the same position. Row copying can reach 100%, but binlog processing remains stuck.

Evidence

The following is an actual checkpoint and surrounding binlog events, with database/table names anonymized and irrelevant columns omitted. Numeric positions are unchanged.

mysql> SELECT gh_ost_chk_id, gh_ost_chk_coords, gh_ost_is_cutover
       FROM app_db._table_a_ghk;

gh_ost_chk_id:     1
gh_ost_chk_coords: mysql-bin-changelog.001644:98771662
gh_ost_is_cutover: 0

mysql> SHOW BINLOG EVENTS
       IN 'mysql-bin-changelog.001644' FROM 98746801 LIMIT 10;

Pos       Event_type      End_log_pos  Info
98746801  Anonymous_Gtid   98746886     SET @@SESSION.GTID_NEXT='ANONYMOUS'
98746886  Query            98746991     BEGIN
98746991  Table_map        98747104     table_id: 5131 (app_db._other_table_gho; another concurrent migration)
98747104  Write_rows       98755290     table_id: 5131
98755290  Write_rows       98763476     table_id: 5131
98763476  Write_rows       98771662     table_id: 5131
98771662  Write_rows       98779848     table_id: 5131  <-- saved checkpoint
98779848  Write_rows       98786279     table_id: 5131 flags: STMT_END_F
98786279  Xid              98786310     COMMIT
98786310  Anonymous_Gtid   98786395     SET @@SESSION.GTID_NEXT='ANONYMOUS'

The checkpoint points to a Write_rows event after its required Table_map, rather than a transaction boundary.

Suspected cause
  • Checkpoint() saves GetCurrentBinlogCoordinates() as LastTrxCoords.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L1812-L1838

  • In file-position mode, currentCoordinates advances on every event, including intermediate row events.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L163-L176

  • Resume loads the saved coordinates and passes that position directly to StartSync(). The new reader therefore lacks the preceding table map.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L580-L595

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L92-L102

The reader separately tracks completed transactions in LastTrxCoords on XID events, but checkpoint creation does not use that value.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L216-L225

Expected behavior

File-position checkpoints should store a restartable transaction boundary, while ensuring all preceding DML has been applied, so --resume can continue.

Vorherrschende Sprache
Go
Sterne
13.6k
Forks
1.4k
Ø Merge
9 T. 19 Std.
Gemergte PRs (30 T.)
1

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus github/gh-ost

Alle Issues in github/gh-ost

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.