File-position checkpoints can point inside a transaction, causing --resume to fail with a missing TABLE_MAP error

Aberta
  • Algo errado?
#1,776 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 10 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
68/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
go, mysql
Domínio
databases

Direção de pesquisa

Start in go/logic/migrator.go at Checkpoint() and resume handling, then trace currentCoordinates, LastTrxCoords, and StartSync() in go/binlog/gomysql_reader.go. Reproduce the file-position checkpoint scenario described in the issue and verify that checkpoints use a restartable transaction boundary while preserving preceding DML, allowing --resume to avoid the missing TABLE_MAP error.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Using gh-ost v1.1.11 (2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa) on Aurora MySQL 8.0 with binlog_format=ROW, without --gtid.

After running with --checkpoint --execute, stopping the process, and restarting with --resume, binlog streaming repeatedly fails with:

StreamEvents encountered unexpected error:
... invalid table id, no corresponding table map event

The streamer repeatedly reconnects at the same position. Row copying can reach 100%, but binlog processing remains stuck.

Evidence

The following is an actual checkpoint and surrounding binlog events, with database/table names anonymized and irrelevant columns omitted. Numeric positions are unchanged.

mysql> SELECT gh_ost_chk_id, gh_ost_chk_coords, gh_ost_is_cutover
       FROM app_db._table_a_ghk;

gh_ost_chk_id:     1
gh_ost_chk_coords: mysql-bin-changelog.001644:98771662
gh_ost_is_cutover: 0

mysql> SHOW BINLOG EVENTS
       IN 'mysql-bin-changelog.001644' FROM 98746801 LIMIT 10;

Pos       Event_type      End_log_pos  Info
98746801  Anonymous_Gtid   98746886     SET @@SESSION.GTID_NEXT='ANONYMOUS'
98746886  Query            98746991     BEGIN
98746991  Table_map        98747104     table_id: 5131 (app_db._other_table_gho; another concurrent migration)
98747104  Write_rows       98755290     table_id: 5131
98755290  Write_rows       98763476     table_id: 5131
98763476  Write_rows       98771662     table_id: 5131
98771662  Write_rows       98779848     table_id: 5131  <-- saved checkpoint
98779848  Write_rows       98786279     table_id: 5131 flags: STMT_END_F
98786279  Xid              98786310     COMMIT
98786310  Anonymous_Gtid   98786395     SET @@SESSION.GTID_NEXT='ANONYMOUS'

The checkpoint points to a Write_rows event after its required Table_map, rather than a transaction boundary.

Suspected cause
  • Checkpoint() saves GetCurrentBinlogCoordinates() as LastTrxCoords.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L1812-L1838

  • In file-position mode, currentCoordinates advances on every event, including intermediate row events.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L163-L176

  • Resume loads the saved coordinates and passes that position directly to StartSync(). The new reader therefore lacks the preceding table map.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/logic/migrator.go#L580-L595

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L92-L102

The reader separately tracks completed transactions in LastTrxCoords on XID events, but checkpoint creation does not use that value.

https://github.com/github/gh-ost/blob/2ed192cd39d8a5173b6ef7c4beaa746a2d3bf1aa/go/binlog/gomysql_reader.go#L216-L225

Expected behavior

File-position checkpoints should store a restartable transaction boundary, while ensuring all preceding DML has been applied, so --resume can continue.

Linguagem predominante
Go
Estrelas
13.6k
Forks
1.4k
Merge médio
9d 19h
PRs com merge (30d)
1

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de github/gh-ost

Todas as issues de github/gh-ost

Issues semelhantes

Mais issues de Go

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.