envoyproxy/envoy

Envoy sometime crash with TTL in xDS

Open

#20,131 建立於 2022年2月28日

在 GitHub 查看
 (4 留言) (0 反應) (1 負責人)C++ (5,373 fork)batch import
area/xdsbughelp wantedinvestigate

倉庫指標

Star
 (27,997 star)
PR 合併指標
 (平均合併 8天) (30 天內合併 378 個 PR)

描述

We use envoy-control-plane to push xDS messages to envoy sidecars in our kubernetes cluster. Sometime envoy sidecar crash with this critical exception - this happens in different PODS with different configuration in different time.

#0  0x00007fa9c74d6817 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00005614aeffd5c6 in Envoy::SignalAction::sigHandler (sig=6, info=0x7fa9c80764b0, context=0x7fa9c8076380) at source/common/signal/signal_action.cc:53
#2  <signal handler called>
#3  0x00007fa9c7111fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007fa9c7113921 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00005614af150f1b in Envoy::TerminateHandler::logOnTerminate() const::$_0::operator()() const (this=<optimized out>)
    at source/exe/terminate_handler.cc:14
#6  0x00005614af150dd9 in Envoy::TerminateHandler::logOnTerminate() const::$_0::__invoke() () at source/exe/terminate_handler.cc:11
#7  0x00005614af601433 in std::__terminate(void (*)()) ()
#8  0x00005614af6013e8 in std::terminate() ()
#9  0x00005614ad438adb in __clang_call_terminate ()
#10 0x00005614aecd2443 in Envoy::Config::TtlManager::ScopedTtlUpdate::~ScopedTtlUpdate (this=this@entry=0x7ffd10346a08) at ./source/common/config/ttl.h:35
#11 0x00005614aecd27a3 in Envoy::Config::TtlManager::TtlManager(std::__1::function<void (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0::operator()() const (this=<optimized out>) at source/common/config/ttl.cc:30
#12 _ZNSt3__18__invokeIRZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS9_ISB_EEEEEEERNS1_5Event10DispatcherERNS1_10TimeSourceEE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSP_DpOSQ_ (__f=...)
    at /opt/llvm/bin/../include/c++/v1/type_traits:3694
#13 std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Config::TtlManager::TtlManager(std::__1::function<void (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0&>(Envoy::Config::TtlManager::TtlManager(std::__1::function<void (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0&) (__args=...)
    at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#14 _ZNSt3__110__function12__alloc_funcIZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENSA_ISC_EEEEEEERNS2_5Event10DispatcherERNS2_10TimeSourceEE3$_0NSA_ISO_EEFvvEEclEv (this=<optimized out>) at /opt/llvm/bin/../include/c++/v1/functional:1558
#15 _ZNSt3__110__function6__funcIZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENSA_ISC_EEEEEEERNS2_5Event10DispatcherERNS2_10TimeSourceEE3$_0NSA_ISO_EEFvvEEclEv (this=<optimized out>) at /opt/llvm/bin/../include/c++/v1/functional:1732
#16 0x00005614aeff1220 in event_process_active_single_queue ()
#17 0x00005614aefeff11 in event_base_loop ()
#18 0x00005614ae85e61c in Envoy::Server::InstanceImpl::run (this=0x54e37fa92f00) at source/server/server.cc:872
#19 0x00005614ad438d64 in Envoy::MainCommonBase::run (this=0x54e37fd0b170) at source/exe/main_common.cc:173
#20 0x00005614ad4395d6 in Envoy::MainCommon::run (this=<optimized out>) at ./source/exe/main_common.h:100
#21 Envoy::MainCommon::main(int, char**, std::__1::function<void (Envoy::Server::Instance&)>) (argc=11, argv=0x7ffd10346ee8, hook=...)
    at source/exe/main_common.cc:256
#22 0x00005614ad43579c in main (argc=2, argv=0x7ffd103464e0) at source/exe/main.cc:24

coredump

envoy version: a9d72603c68da3a10a1c0d021d01c7877e6f2a30/1.21.0/Clean/RELEASE/BoringSS

we turn of TTL on xDS messages and envoy stops crashing

貢獻者指南