yugabyte/yugabyte-db

[docdb] Segfault on bootstrap if filesystem does not support fallocate

Open

#3871 aperta il 6 mar 2020

Vedi su GitHub
 (0 commenti) (0 reazioni) (0 assegnatari)C (1003 fork)batch import
area/docdbgood first issuehelp wantedkind/enhancementpriority/medium

Metriche repository

Star
 (8229 star)
Metriche merge PR
 (Merge medio 17g 21h) (92 PR mergiate in 30 g)

Descrizione

Jira Link: DB-1810 Stack trace from gdb

#0  IsAllZeros (s=...) at ../../../../../src/yb/consensus/log_util.cc:344
#1  yb::log::ReadableLogSegment::ScanForValidEntryHeaders (this=this@entry=0x2e23680, offset=162, has_valid_entries=has_valid_entries@entry=0x7efe0664ebf0) at ../../../../../src/yb/consensus/log_util.cc:679
#2  0x00007efe1db3c98d in yb::log::ReadableLogSegment::ReadFirstEntryMetadata (this=0x2e23680) at ../../../../../src/yb/consensus/log_util.cc:621
#3  0x00007efe1e6f5c63 in yb::tablet::TabletBootstrap::PlaySegments (this=this@entry=0x7efe0664f790, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:871
#4  0x00007efe1e6f7780 in yb::tablet::TabletBootstrap::Bootstrap (this=this@entry=0x7efe0664f790, rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:382
#5  0x00007efe1e6fcb94 in yb::tablet::BootstrapTablet (data=..., rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap_if.cc:89
#6  0x00007efe1f44ba4f in yb::master::SysCatalogTable::OpenTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:512
#7  0x00007efe1f44ce90 in yb::master::SysCatalogTable::SetupTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:473
#8  0x00007efe1f44d69a in yb::master::SysCatalogTable::Load (this=0x2b1c380, fs_manager=0x29f4780) at ../../../../../src/yb/master/sys_catalog.cc:255
#9  0x00007efe1f37772e in yb::master::CatalogManager::InitSysCatalogAsync (this=this@entry=0x2d5a000, is_first_run=is_first_run@entry=false) at ../../../../../src/yb/master/catalog_manager.cc:1252
#10 0x00007efe1f383f29 in yb::master::CatalogManager::Init (this=0x2d5a000, is_first_run=<optimized out>) at ../../../../../src/yb/master/catalog_manager.cc:517
#11 0x00007efe1f41c50a in yb::master::Master::InitCatalogManager (this=this@entry=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:250
#12 0x00007efe1f41c616 in yb::master::Master::InitCatalogManagerTask (this=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:239
#13 0x00007efe161f64c4 in yb::ThreadPool::DispatchThread (this=0x2d60a00, permanent=false) at ../../../../../src/yb/util/threadpool.cc:608
#14 0x00007efe161f2bcf in operator() (this=0x318a058) at /tmp/testing/yugabyte-2.1.1.0/linuxbrew-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:2267
#15 yb::Thread::SuperviseThread (arg=0x318a000) at ../../../../../src/yb/util/thread.cc:744
#16 0x00007efe11bb4694 in start_thread (arg=0x7efe06651700) at pthread_create.c:333
#17 0x00007efe112f141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Setup, with my home dir on NFS:

bin/yb-ctl --rf 3 create
echo "create database foo;" | bin/ysqlsh
bin/yb-ctl stop
bin/yb-ctl start
# ^^^ this fails, all masters segfault

Turns out the logs were showing this env_posix warning: The filesystem does not support fallocate().

Probably the log reader code assumes it can actually read at some offset in the WAL files, which might be fine, if the file was preallocated...

Given this error showed up much later, it means our code definitely makes assumptions on fallocate working, so we shouldn't have this as just a WARNING but instead fail early... a) turn the WARNINGS into fatals b) add this check to the very top-level (in the init phase in main), so we can fail asap and notify the user of a misconfigured setup

cc @mbautin

Guida contributor