oceanbase/miniob

[BUG] miniob crashes when concurrently creating tables

Open

#355 opened on Mar 22, 2024

View on GitHub
 (10 comments) (0 reactions) (1 assignee)C++ (1,622 forks)auto 404
bughelp wanted

Repository metrics

Stars
 (4,387 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the bug A clear and concise description of what the bug is.

** Environment** Environment Details sometimes important

  • OS Version: Linux 10-2-12-124 5.14.0-284.18.1.el9_2.x86_64
  • CPU Arch(x86/arm): x86
  • Compiler: clang version 13.0.0
  • Others:

Fast Reproduce Steps(Required) Steps to reproduce the behavior:

  1. setup miniob
  2. concurrently send insert requests to miniob with 20 threads, and each thread send 10w rows.

Expected behavior A clear and concise description of what you expected to happen.

no crash

Actual Behavior What is the result? picture is allowed

crash

Additional context Add any other context about the problem here.

    #0 0x5fddf1 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, 
std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> > >::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__hash_table:2116:34
    #1 0x5fd1de in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, 
std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> > >::__emplace_unique_extract_key<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> >(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&, std::__1::__extract_key_first_tag) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__hash_table:1115:14
#2 0x5fcfcf in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, 
std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> > >::__emplace_unique<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> >(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__hash_table:1079:14
    #3 0x5fcbbf in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, 
std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> > >::__insert_unique<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void>(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__hash_table:1137:14
    #4 0x5e206d in std::__1::pair<std::__1::__hash_map_iterator<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*> >, bool> std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, DiskBufferPool*> > >::insert<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void>(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/unordered_map:1141:30
    #5 0x5cf13e in BufferPoolManager::open_file(char const*, DiskBufferPool*&) /DATA/disk3/xzx/miniob/src/observer/storage/buffer/disk_buffer_pool.cpp:688:17
    #6 0x68fdb3 in Table::init_record_handler(char const*) /DATA/disk3/xzx/miniob/src/observer/storage/table/table.cpp:306:41
    #7 0x68e6f7 in Table::create(int, char const*, char const*, char const*, int, AttrInfoSqlNode const*) /DATA/disk3/xzx/miniob/src/observer/storage/table/table.cpp:113:8
    #8 0x623ff4 in Db::create_table(char const*, int, AttrInfoSqlNode const*) /DATA/disk3/xzx/miniob/src/observer/storage/db/db.cpp:92:15
    #9 0x4a519d in CreateTableExecutor::execute(SQLStageEvent*) /DATA/disk3/xzx/miniob/src/observer/sql/executor/create_table_executor.cpp:37:38
    #10 0x49f420 in CommandExecutor::execute(SQLStageEvent*) /DATA/disk3/xzx/miniob/src/observer/sql/executor/command_executor.cpp:41:23
    #11 0x49bbfb in ExecuteStage::handle_request(SQLStageEvent*) /DATA/disk3/xzx/miniob/src/observer/sql/executor/execute_stage.cpp:46:27
    #12 0x4622ad in SqlTaskHandler::handle_sql(SQLStageEvent*) /DATA/disk3/xzx/miniob/src/observer/net/sql_task_handler.cpp:80:23
    #13 0x45f018 in SqlTaskHandler::handle_event(Communicator*) /DATA/disk3/xzx/miniob/src/observer/net/sql_task_handler.cpp:37:9
    #14 0x4712de in Worker::operator()() /DATA/disk3/xzx/miniob/src/observer/net/one_thread_per_connection_thread_handler.cpp:95:29
    #15 0x46e874 in decltype(static_cast<Worker&>(fp)()) std::__1::__invoke<Worker&>(Worker&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/type_traits:3918:1
    #16 0x46e84c in std::__1::__invoke_of<Worker&>::type std::__1::reference_wrapper<Worker>::operator()<>() const /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__functional/reference_wrapper.h:69:16
    #17 0x46e7f4 in decltype(static_cast<std::__1::reference_wrapper<Worker>>(fp)()) std::__1::__invoke<std::__1::reference_wrapper<Worker> >(std::__1::reference_wrapper<Worker>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/type_traits:3918:1
    #18 0x46e794 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::reference_wrapper<Worker> >(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::reference_wrapper<Worker> >&, std::__1::__tuple_indices<>) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/thread:280:5
    #19 0x46de2f in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::reference_wrapper<Worker> > >(void*) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/thread:291:5
    #20 0x7f5903c9f801 in start_thread (/usr/lib64/libc.so.6+0x9f801)
    #21 0x7f5903c3f44f in __GI___clone3 (/usr/lib64/libc.so.6+0x3f44f)


SUMMARY: AddressSanitizer: SEGV /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__hash_table:2116:34 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, void*>*>, bool> 
std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> > >::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DiskBufferPool*>&&)

    #0 0x3a1cbc in pthread_create /root/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:207:3
    #1 0x46dd18 in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/__threading_support:514:10
    #2 0x46d568 in std::__1::thread::thread<std::__1::reference_wrapper<Worker>, void>(std::__1::reference_wrapper<Worker>&&) /DATA/disk3/xzx/env-tiflash/sysroot/bin/../include/c++/v1/thread:307:16
    #3 0x46c2bd in Worker::start() /DATA/disk3/xzx/miniob/src/observer/net/one_thread_per_connection_thread_handler.cpp:43:19
    #4 0x468ab6 in OneThreadPerConnectionThreadHandler::new_connection(Communicator*) /DATA/disk3/xzx/miniob/src/observer/net/one_thread_per_connection_thread_handler.cpp:132:18
    #5 0x40cf7e in NetServer::accept(int) /DATA/disk3/xzx/miniob/src/observer/net/server.cpp:129:25
    #6 0x419cc5 in NetServer::serve() /DATA/disk3/xzx/miniob/src/observer/net/server.cpp:285:13
    #7 0x3ef478 in main /DATA/disk3/xzx/miniob/src/observer/main.cpp:196:13
    #8 0x7f5903c3feaf in __libc_start_call_main (/usr/lib64/libc.so.6+0x3feaf)

Contributor guide

Research direction
Investigate the concurrent table creation code path and identify missing synchronization in BufferPoolManager::open file. Check for race conditions in the unordered map insertion.
Tech stack
cpp
Domain
database
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++MultithreadingConcurrency
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35