yugabyte/yugabyte-db

[docdb] Segfault on bootstrap if filesystem does not support fallocate

Open

#3,871 创建于 2020年3月6日

在 GitHub 查看
 (0 评论) (0 反应) (0 负责人)C (1,003 fork)batch import
area/docdbgood first issuehelp wantedkind/enhancementpriority/medium

仓库指标

Star
 (8,229 star)
PR 合并指标
 (平均合并 17天 21小时) (30 天内合并 92 个 PR)

描述

Jira Link: DB-1810 Stack trace from gdb

#0  IsAllZeros (s=...) at ../../../../../src/yb/consensus/log_util.cc:344
#1  yb::log::ReadableLogSegment::ScanForValidEntryHeaders (this=this@entry=0x2e23680, offset=162, has_valid_entries=has_valid_entries@entry=0x7efe0664ebf0) at ../../../../../src/yb/consensus/log_util.cc:679
#2  0x00007efe1db3c98d in yb::log::ReadableLogSegment::ReadFirstEntryMetadata (this=0x2e23680) at ../../../../../src/yb/consensus/log_util.cc:621
#3  0x00007efe1e6f5c63 in yb::tablet::TabletBootstrap::PlaySegments (this=this@entry=0x7efe0664f790, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:871
#4  0x00007efe1e6f7780 in yb::tablet::TabletBootstrap::Bootstrap (this=this@entry=0x7efe0664f790, rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap.cc:382
#5  0x00007efe1e6fcb94 in yb::tablet::BootstrapTablet (data=..., rebuilt_tablet=rebuilt_tablet@entry=0x7efe0664fa50, rebuilt_log=rebuilt_log@entry=0x7efe0664fa20, consensus_info=consensus_info@entry=0x7efe0664fb50) at ../../../../../src/yb/tablet/tablet_bootstrap_if.cc:89
#6  0x00007efe1f44ba4f in yb::master::SysCatalogTable::OpenTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:512
#7  0x00007efe1f44ce90 in yb::master::SysCatalogTable::SetupTablet (this=this@entry=0x2b1c380, metadata=...) at ../../../../../src/yb/master/sys_catalog.cc:473
#8  0x00007efe1f44d69a in yb::master::SysCatalogTable::Load (this=0x2b1c380, fs_manager=0x29f4780) at ../../../../../src/yb/master/sys_catalog.cc:255
#9  0x00007efe1f37772e in yb::master::CatalogManager::InitSysCatalogAsync (this=this@entry=0x2d5a000, is_first_run=is_first_run@entry=false) at ../../../../../src/yb/master/catalog_manager.cc:1252
#10 0x00007efe1f383f29 in yb::master::CatalogManager::Init (this=0x2d5a000, is_first_run=<optimized out>) at ../../../../../src/yb/master/catalog_manager.cc:517
#11 0x00007efe1f41c50a in yb::master::Master::InitCatalogManager (this=this@entry=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:250
#12 0x00007efe1f41c616 in yb::master::Master::InitCatalogManagerTask (this=0x7fff4c38f480) at ../../../../../src/yb/master/master.cc:239
#13 0x00007efe161f64c4 in yb::ThreadPool::DispatchThread (this=0x2d60a00, permanent=false) at ../../../../../src/yb/util/threadpool.cc:608
#14 0x00007efe161f2bcf in operator() (this=0x318a058) at /tmp/testing/yugabyte-2.1.1.0/linuxbrew-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:2267
#15 yb::Thread::SuperviseThread (arg=0x318a000) at ../../../../../src/yb/util/thread.cc:744
#16 0x00007efe11bb4694 in start_thread (arg=0x7efe06651700) at pthread_create.c:333
#17 0x00007efe112f141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Setup, with my home dir on NFS:

bin/yb-ctl --rf 3 create
echo "create database foo;" | bin/ysqlsh
bin/yb-ctl stop
bin/yb-ctl start
# ^^^ this fails, all masters segfault

Turns out the logs were showing this env_posix warning: The filesystem does not support fallocate().

Probably the log reader code assumes it can actually read at some offset in the WAL files, which might be fine, if the file was preallocated...

Given this error showed up much later, it means our code definitely makes assumptions on fallocate working, so we shouldn't have this as just a WARNING but instead fail early... a) turn the WARNINGS into fatals b) add this check to the very top-level (in the init phase in main), so we can fail asap and notify the user of a misconfigured setup

cc @mbautin

贡献者指南