Migration hangs when --attempt-instant-ddl succeeds

Aperta
  • Qualcosa non va?
#1,795 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 10 giorni

@d-bytebase ci sta già lavorando.

Dal 2/11/2026.

  • #1796 di @d-bytebase — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, mysql
Ambito
databases

Direzione di ricerca

Start with Migrate() in go/logic/migrator.go, where the instant-DDL branch returns without reading ghostTableMigrated, then follow onChangelogStateEvent and notifyListeners to see where listenersMutex is held during the blocking send. The binlog side is shouldDecodeRowsEvent in the event streamer, reached through BinlogSyncer.Close() in finalCleanup. Done means the instant-DDL success path returns without hanging, with a test covering that short-circuit.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Overview

A migration run with --attempt-instant-ddl against a server that can apply the
ALTER instantly applies it, logs Success! table ... migrated instantly, and then
never returns. gh-ost hangs until it is killed.

Reproduced on master (dd44453f) and on v1.1.11.

Why

Migrate() short-circuits on the instant-DDL path: on success it calls
finalCleanup(), runs the OnSuccess hook and returns, so it never reads
<-mgtr.ghostTableMigrated.

https://github.com/github/gh-ost/blob/dd44453f489bdeb8bee5b93194a0a481e001e598/go/logic/migrator.go#L534-L552

Three goroutines then form a cycle:

  1. The changelog listener parks on the GhostTableMigrated send in
    onChangelogStateEvent. ghostTableMigrated is unbuffered and the
    instant-DDL path never reads it, so the send blocks. It is wrapped in
    base.SendWithContext, which only unblocks on context cancellation — and
    nothing cancels the context here, precisely because Migrate() has not
    returned. The listener is invoked from notifyListeners, which holds
    listenersMutex across the synchronous call, so the mutex stays held for as
    long as the send is parked.

  2. The binlog parser goroutine (go-mysql's BinlogSyncer.onStream) calls
    EventsStreamer.shouldDecodeRowsEvent through RowsEventDecodeFunc and
    blocks taking that same listenersMutex.

  3. BinlogSyncer.Close(), reached from finalCleanup() →
    EventsStreamer.Close(), ends in b.wg.Wait() and waits for that parser
    goroutine, which is never coming back.

Step 2 is what makes the cycle reachable. shouldDecodeRowsEvent was introduced
by #1699 ("Don't parse binlog events of unrelated tables", first released in
v1.1.11); it put listenersMutex on the binlog parser's goroutine, so a listener
parked while holding that mutex now stalls the parser directly, and with it
BinlogSyncer.Close().

--attempt-instant-ddl defaults to off and the short-circuit has no test
coverage, which is presumably why this has not been reported before. We hit it in
a downstream consumer that passes the flag for every migration, where every ALTER
MySQL could apply instantly hung.

Reproduction

Any instant-applicable ALTER on MySQL 8.0 with the flag set, e.g.

gh-ost --host=... --database=test --table=testing \
  --alter='ADD COLUMN author varchar(255)' \
  --attempt-instant-ddl --allow-on-master --execute

The column is added, Success! ... migrated instantly is logged, and the process
does not exit.

Fix

PR to follow: scope listenersMutex to the listener lookup in notifyListeners,
and give ghostTableMigrated a one-slot buffer.

Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
9g 19h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/gh-ost

Tutte le issue di github/gh-ost

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.