yugabyte/yugabyte-db

[docdb] Segfault on bootstrap if filesystem does not support fallocate

Open

#3 871 ouverte le 6 mars 2020

Voir sur GitHub
 (0 commentaires) (0 réactions) (0 assignés)C (1 003 forks)batch import
area/docdbgood first issuehelp wantedkind/enhancementpriority/medium

Métriques du dépôt

Stars
 (8 229 stars)
Métriques de merge PR
 (Merge moyen 17j 21h) (92 PRs mergées en 30 j)

Description

Jira Link: DB-1810 Stack trace from gdb

#0  IsAllZeros (s=...) at ../../../../../src/yb/consensus/log_util.cc:344
#1  yb::log::ReadableLogSegment::ScanForValidEntryHeaders (this=this@entry=0x2e23680, offset=162, has_valid_entries=has_valid_entries@entry=0x7efe0664ebf0) at ../../../../../src/yb/consensus/log_util.cc:679
#2  0x00007efe1db3c98d in yb::log::ReadableLogSegment::ReadFirstEntryMetadata (this=0x2e23680) at ../../../../../src/yb/consensus/log_util.cc:621
#3  0x00007efe1e6f5c63 in yb::tablet::TabletBootstrap::PlaySegments (this=this@entry=0x7efe0664f790, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:871
#4  0x00007efe1e6f7780 in yb::tablet::TabletBootstrap::Bootstrap (this=this@entry=0x7efe0664f790, rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:382
#5  0x00007efe1e6fcb94 in yb::tablet::BootstrapTablet (data=..., rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap_if.cc:89
#6  0x00007efe1f44ba4f in yb::master::SysCatalogTable::OpenTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:512
#7  0x00007efe1f44ce90 in yb::master::SysCatalogTable::SetupTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:473
#8  0x00007efe1f44d69a in yb::master::SysCatalogTable::Load (this=0x2b1c380, fs_manager=0x29f4780) at ../../../../../src/yb/master/sys_catalog.cc:255
#9  0x00007efe1f37772e in yb::master::CatalogManager::InitSysCatalogAsync (this=this@entry=0x2d5a000, is_first_run=is_first_run@entry=false) at ../../../../../src/yb/master/catalog_manager.cc:1252
#10 0x00007efe1f383f29 in yb::master::CatalogManager::Init (this=0x2d5a000, is_first_run=<optimized out>) at ../../../../../src/yb/master/catalog_manager.cc:517
#11 0x00007efe1f41c50a in yb::master::Master::InitCatalogManager (this=this@entry=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:250
#12 0x00007efe1f41c616 in yb::master::Master::InitCatalogManagerTask (this=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:239
#13 0x00007efe161f64c4 in yb::ThreadPool::DispatchThread (this=0x2d60a00, permanent=false) at ../../../../../src/yb/util/threadpool.cc:608
#14 0x00007efe161f2bcf in operator() (this=0x318a058) at /tmp/testing/yugabyte-2.1.1.0/linuxbrew-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:2267
#15 yb::Thread::SuperviseThread (arg=0x318a000) at ../../../../../src/yb/util/thread.cc:744
#16 0x00007efe11bb4694 in start_thread (arg=0x7efe06651700) at pthread_create.c:333
#17 0x00007efe112f141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Setup, with my home dir on NFS:

bin/yb-ctl --rf 3 create
echo "create database foo;" | bin/ysqlsh
bin/yb-ctl stop
bin/yb-ctl start
# ^^^ this fails, all masters segfault

Turns out the logs were showing this env_posix warning: The filesystem does not support fallocate().

Probably the log reader code assumes it can actually read at some offset in the WAL files, which might be fine, if the file was preallocated...

Given this error showed up much later, it means our code definitely makes assumptions on fallocate working, so we shouldn't have this as just a WARNING but instead fail early... a) turn the WARNINGS into fatals b) add this check to the very top-level (in the init phase in main), so we can fail asap and notify the user of a misconfigured setup

cc @mbautin

Guide contributeur