File-position checkpoints can point inside a transaction, causing --resume to fail with a missing TABLE_MAP error

Abierto
  • ¿Algo no está bien?
#1,776 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 10 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go, mysql
Área
databases

Línea de trabajo

Start in go/logic/migrator.go at Checkpoint() and resume handling, then trace currentCoordinates, LastTrxCoords, and StartSync() in go/binlog/gomysql_reader.go. Reproduce the file-position checkpoint scenario described in the issue and verify that checkpoints use a restartable transaction boundary while preserving preceding DML, allowing --resume to avoid the missing TABLE_MAP error.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Using gh-ost v1.1.11 (2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa) on Aurora MySQL 8.0 with binlog_format=ROW, without --gtid.

After running with --checkpoint --execute, stopping the process, and restarting with --resume, binlog streaming repeatedly fails with:

StreamEvents encountered unexpected error:
... invalid table id, no corresponding table map event

The streamer repeatedly reconnects at the same position. Row copying can reach 100%, but binlog processing remains stuck.

Evidence

The following is an actual checkpoint and surrounding binlog events, with database/table names anonymized and irrelevant columns omitted. Numeric positions are unchanged.

mysql> SELECT gh_ost_chk_id, gh_ost_chk_coords, gh_ost_is_cutover
       FROM app_db._table_a_ghk;

gh_ost_chk_id:     1
gh_ost_chk_coords: mysql-bin-changelog.001644:98771662
gh_ost_is_cutover: 0

mysql> SHOW BINLOG EVENTS
       IN 'mysql-bin-changelog.001644' FROM 98746801 LIMIT 10;

Pos       Event_type      End_log_pos  Info
98746801  Anonymous_Gtid   98746886     SET @@SESSION.GTID_NEXT='ANONYMOUS'
98746886  Query            98746991     BEGIN
98746991  Table_map        98747104     table_id: 5131 (app_db._other_table_gho; another concurrent migration)
98747104  Write_rows       98755290     table_id: 5131
98755290  Write_rows       98763476     table_id: 5131
98763476  Write_rows       98771662     table_id: 5131
98771662  Write_rows       98779848     table_id: 5131  <-- saved checkpoint
98779848  Write_rows       98786279     table_id: 5131 flags: STMT_END_F
98786279  Xid              98786310     COMMIT
98786310  Anonymous_Gtid   98786395     SET @@SESSION.GTID_NEXT='ANONYMOUS'

The checkpoint points to a Write_rows event after its required Table_map, rather than a transaction boundary.

Suspected cause
  • Checkpoint() saves GetCurrentBinlogCoordinates() as LastTrxCoords.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L1812-L1838

  • In file-position mode, currentCoordinates advances on every event, including intermediate row events.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L163-L176

  • Resume loads the saved coordinates and passes that position directly to StartSync(). The new reader therefore lacks the preceding table map.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L580-L595

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L92-L102

The reader separately tracks completed transactions in LastTrxCoords on XID events, but checkpoint creation does not use that value.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L216-L225

Expected behavior

File-position checkpoints should store a restartable transaction boundary, while ensuring all preceding DML has been applied, so --resume can continue.

Lenguaje dominante
Go
Estrellas
13.6k
Forks
1.4k
Merge medio
9 d 19 h
PR fusionados (30 d)
1

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de github/gh-ost

Todos los issues de github/gh-ost

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.