TiFlash v8.5.8: SIGSEGV in Block::cloneEmptyColumns while flushing a ColumnFileInMemory

Aperta
#11,235 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, databases

Direzione di ricerca

Start with dbms/src/Core/Block.cpp around line 373 and trace how Block slots are created or mutated before ColumnFileInMemory::readDataForFlush calls cloneEmptyColumns(). Instrument the slot’s column and type pointers along that path to find where both become unset. Done means identifying the creation or mutation path that produces the invalid slot; the issue provides no standalone reproducer or named test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

contribution

TiFlash crashed while flushing a Region on a production cluster. The application signal log reported a read from a null address. The service manager restarted TiFlash after the crash.

Environment: TiFlash v8.5.8, engine Git hash 3abdec8bff315647a056e6bf910abd4b59af2011, Linux x86-64.

Crash stack (caller → callee, selected frames):

TiFlashObserver::pre_exec_admin
  → TryFlushData
  → KVStore::tryFlushRegionData
  → KVStore::forceFlushRegionDataImpl
  → Segment::flushCache
  → DeltaValueSpace::flush
  → MemTableSet::buildFlushTask
  → ColumnFileInMemory::readDataForFlush
  → Block::cloneEmptyColumns() const

Evidence from the faulting thread: The application stack and core identify the same instruction offset, tiflash+0x782ea24. GDB resolved it to Block::cloneEmptyColumns() const+244, at dbms/src/Core/Block.cpp:373. The relevant source expression is:

columns[i] = data[i].column ? data[i].column->cloneEmpty() : data[i].type->createColumn();

In the disassembly, the null check on column took the fallback branch. That branch loaded type into %rsi. At the faulting instruction, mov (%rsi),%rax, %rsi was zero. The loop index was 8. This indicates that both pointers read from this in-memory Block slot were null. The crash occurred before the virtual call to createColumn().

Diagnostic limits: The core was truncated. GDB warned of a possible executable mismatch, and the TiFlash executable had no build ID. The faulting instruction offset agrees with the application stack, but the core did not retain enough accessible memory to inspect the slot independently or recover the Region ID. We do not know how the Block reached this state.

Reproduction: Observed in production. We do not have a standalone reproducer.

Is there a known issue in this version that can leave a ColumnFileInMemory Block slot with both column and type unset? Which creation or mutation path would you instrument to identify when the slot first enters this state?

Lingua principale
C++
Stelle
1k
Fork
428
Merge medio
2g 7h
PR unite (30g)
100

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pingcap/tiflash

Tutte le issue di pingcap/tiflash

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.