envoyproxy/envoy

another envoy shut down issue: assert numSymbols() == 0

Open

#15,180 opened on 2021年2月24日

GitHub で見る
 (8 comments) (0 reactions) (1 assignee)C++ (5,373 forks)batch import
area/statsbughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (平均マージ 8d) (30d で 378 merged PRs)

説明

The signal-unsafe can be ignored for now. I paste for keep the context self-contained.

The broken assertion is probably '#18'.

My guess is that this is wasm specific issue.

    #18 Envoy::Stats::SymbolTableImpl::~SymbolTableImpl() <null> (envoy+0xd128b9f)
    #19 Envoy::MainCommonBase::~MainCommonBase() <null> (envoy+0x5a5e8c7)
    #20 Envoy::MainCommon::~MainCommon() <null> (envoy+0x5a5e789)
    #21 std::__1::default_delete<Envoy::MainCommon>::operator()(Envoy::MainCommon*) const <null> (envoy+0x5a5e716)
    #22 std::__1::unique_ptr<Envoy::MainCommon, std::__1::default_delete<Envoy::MainCommon> >::reset(Envoy::MainCommon*) <null> (envoy+0x5a5e640
2021/02/24 21:16:04 stop envoy ...
2021/02/24 21:16:04 HTTP POST http://127.0.0.1:20181/quitquitquit
[2021-02-24 21:16:04.216][29505][info][main] [external/envoy/source/server/server.cc:821] shutting down server instance
[2021-02-24 21:16:04.216][29505][info][main] [external/envoy/source/server/server.cc:873] Notifying 0 callback(s) with completion.
[2021-02-24 21:16:04.218][29505][info][main] [external/envoy/source/server/server.cc:767] main dispatch loop exited
[2021-02-24 21:16:04.243][29505][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 13, 
[2021-02-24 21:16:04.245][29505][info][main] [external/envoy/source/server/server.cc:813] exiting
2021/02/24 21:16:04 got metrics 2, want 4
2021/02/24 21:16:04 sleeping till next check
[2021-02-24 21:16:04.327][29505][critical][assert] [external/envoy/source/common/stats/symbol_table_impl.cc:194] assert failure: numSymbols() == 0.
[2021-02-24 21:16:04.327][29505][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x7341
[2021-02-24 21:16:04.327][29505][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
2021/02/24 21:16:05 got metrics 2, want 4
2021/02/24 21:16:05 sleeping till next check
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=29505)
    #0 malloc <null> (envoy+0x5356884)
    #1 operator new(unsigned long) <null> (libc++.so.1+0x86084)
    #2 std::__1::allocator<char>::allocate(unsigned long, void const*) <null> (envoy+0x53cb6b4)
    #3 std::__1::allocator_traits<std::__1::allocator<char> >::allocate(std::__1::allocator<char>&, unsigned long) <null> (envoy+0x53cb42a)
    #4 fmt::v7::basic_memory_buffer<char, 250ul, std::__1::allocator<char> >::grow(unsigned long) <null> (envoy+0x53caf41)
    #5 fmt::v7::detail::buffer<char>::reserve(unsigned long) <null> (envoy+0x53cefc0)
    #6 void fmt::v7::detail::buffer<char>::append<char>(char const*, char const*) <null> (envoy+0x53ced52)
    #7 spdlog::details::fmt_helper::append_string_view(fmt::v7::basic_string_view<char>, fmt::v7::basic_memory_buffer<char, 250ul, std::__1::allocator<char> >&) <null> (envoy+0x53ce79e)
    #8 spdlog::details::v_formatter<spdlog::details::null_scoped_padder>::format(spdlog::details::log_msg const&, tm const&, fmt::v7::basic_memory_buffer<char, 250ul, std::__1::allocator<char> >&) <null> (envoy+0x543d3bf)
    #9 spdlog::pattern_formatter::format(spdlog::details::log_msg const&, fmt::v7::basic_memory_buffer<char, 250ul, std::__1::allocator<char> >&) <null> (envoy+0x53c9947)
    #10 Envoy::Logger::DelegatingLogSink::log(spdlog::details::log_msg const&) <null> (envoy+0xec120dc)
    #11 spdlog::logger::sink_it_(spdlog::details::log_msg const&) <null> (envoy+0x53c91db)
    #12 spdlog::logger::log_it_(spdlog::details::log_msg const&, bool, bool) <null> (envoy+0x589141a)
    #13 void spdlog::logger::log_<fmt::v7::basic_string_view<char>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(spdlog::source_loc, spdlog::level::level_enum, fmt::v7::basic_string_view<char> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (envoy+0x58bd7e9)
    #14 void spdlog::logger::log<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(spdlog::source_loc, spdlog::level::level_enum, fmt::v7::basic_string_view<char>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (envoy+0x58ba891)
    #15 Envoy::BackwardsTrace::logTrace() <null> (envoy+0xcc20d78)
    #16 Envoy::SignalAction::sigHandler(int, siginfo_t*, void*) <null> (envoy+0xcc1d532)
    #17 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) <null> (envoy+0x535e9ef)
    #18 Envoy::Stats::SymbolTableImpl::~SymbolTableImpl() <null> (envoy+0xd128b9f)
    #19 Envoy::MainCommonBase::~MainCommonBase() <null> (envoy+0x5a5e8c7)
    #20 Envoy::MainCommon::~MainCommon() <null> (envoy+0x5a5e789)
    #21 std::__1::default_delete<Envoy::MainCommon>::operator()(Envoy::MainCommon*) const <null> (envoy+0x5a5e716)
    #22 std::__1::unique_ptr<Envoy::MainCommon, std::__1::default_delete<Envoy::MainCommon> >::reset(Envoy::MainCommon*) <null> (envoy+0x5a5e640)
    #23 std::__1::unique_ptr<Envoy::MainCommon, std::__1::default_delete<Envoy::MainCommon> >::~unique_ptr() <null> (envoy+0x5a3783c)
    #24 Envoy::MainCommon::main(int, char**, std::__1::function<void (Envoy::Server::Instance&)>) <null> (envoy+0x5a312ee)
    #25 main <null> (envoy+0x5a2d6f5)

コントリビューターガイド