envoyproxy/envoy

Crash in Envoy::Http::AsyncStreamImpl::closeRemote on shutdown

Open

#7,983 opened on Aug 20, 2019

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
bughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Seeing the following crash running off 38a4d86407668612ccbc4cba65d11be6cbd81b32:

2019-08-20_22:24:17.36421 [2019-08-20 22:24:17.364][374905][warning][main] [external/envoy/source/server/server.cc:469] caught SIGTERM
2019-08-20_22:24:17.36423 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:574] shutting down server instance
2019-08-20_22:24:17.36424 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:528] main dispatch loop exited
2019-08-20_22:24:17.36959 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation f
ault, suspect faulting address 0x0
2019-08-20_22:24:17.36962 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/
stack_decode.py to get line numbers):
2019-08-20_22:24:17.37594 [2019-08-20 22:24:17.375][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7
f1f290895d0]
2019-08-20_22:24:17.39615 [2019-08-20 22:24:17.396][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Http::AsyncStreamImpl::closeRemote() [0x8f0b46]
2019-08-20_22:24:17.40303 [2019-08-20 22:24:17.402][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Http::AsyncStreamImpl::encodeData() [0x8f5148]
2019-08-20_22:24:17.41097 [2019-08-20 22:24:17.410][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Http::Utility::sendLocalReply() [0xa05ff8]
2019-08-20_22:24:17.41454 [2019-08-20 22:24:17.414][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Http::AsyncStreamImpl::sendLocalReply() [0x8f0e69]
2019-08-20_22:24:17.41906 [2019-08-20 22:24:17.419][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Router::Filter::onUpstreamAbort() [0x8f71c3]
2019-08-20_22:24:17.42254 [2019-08-20 22:24:17.422][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Router::Filter::onUpstreamReset() [0x902bee]
2019-08-20_22:24:17.42603 [2019-08-20 22:24:17.425][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Router::Filter::UpstreamRequest::onResetStream() [0x902fb2]
2019-08-20_22:24:17.42950 [2019-08-20 22:24:17.429][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::Http1::ClientConnectionImpl::onResetStream() [0x95f30d]
2019-08-20_22:24:17.43457 [2019-08-20 22:24:17.434][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::CodecClient::onEvent() [0x8e27a4]
2019-08-20_22:24:17.44230 [2019-08-20 22:24:17.442][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Network::ConnectionImpl::raiseEvent() [0x753b25]
2019-08-20_22:24:17.44620 [2019-08-20 22:24:17.446][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Network::ConnectionImpl::closeSocket() [0x753045]
2019-08-20_22:24:17.45422 [2019-08-20 22:24:17.454][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Network::ConnectionImpl::close() [0x75489d]
2019-08-20_22:24:17.45843 [2019-08-20 22:24:17.458][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http1::ConnPoolImpl::~ConnPoolImpl() [0x88abf5]
2019-08-20_22:24:17.46249 [2019-08-20 22:24:17.462][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::Http1::ProdConnPoolImpl::~ProdConnPoolImpl() [0x88ad00]
2019-08-20_22:24:17.46651 [2019-08-20 22:24:17.466][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Upstream::ConnPoolMap<>::clearActivePools() [0x78500e]
2019-08-20_22:24:17.46989 [2019-08-20 22:24:17.469][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Upstream::ConnPoolMap<>::~ConnPoolMap() [0x78c2e5]
2019-08-20_22:24:17.47316 [2019-08-20 22:24:17.473][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x78c428]
2019-08-20_22:24:17.47640 [2019-08-20 22:24:17.476][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: std::_Hashtable<>::clear() [0x784e4a]
2019-08-20_22:24:17.47980 [2019-08-20 22:24:17.479][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::~ThreadLocalClusterManagerImpl() [0x789c2b]
2019-08-20_22:24:17.48601 [2019-08-20 22:24:17.485][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::ThreadLoc
al::InstanceImpl::shutdownThread() [0x70873a]
2019-08-20_22:24:17.48990 [2019-08-20 22:24:17.489][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Server::W
orkerImpl::threadRoutine() [0x746d49]
2019-08-20_22:24:17.49403 [2019-08-20 22:24:17.493][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Thread::T
hreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0xc879c5]
2019-08-20_22:24:17.49410 [2019-08-20 22:24:17.494][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: start_thread [0x
7f1f29081dd5]
2019-08-20_22:24:17.49412 AsyncClient 0x4981808, stream_id_: 1003858367269869698
2019-08-20_22:24:17.49413 &stream_info_:
2019-08-20_22:24:17.49413   StreamInfoImpl 0x4981960, protocol_: 1, response_code_: null, response_code_details_: upstream_reset_before_response_started{connection termination}, health_check_request_: 0, rout
e_name_:

Not 100% sure that this was introduced with our recent update, as we've been seeing crashes with similar traces on older versions.

Contributor guide

Research direction
Investigate the null pointer dereference in AsyncStreamImpl::closeRemote during shutdown. Trace the call path from the backtrace to determine why the stream object is in an inconsistent state when the connection pool is destroyed.
Tech stack
cpp
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++EnvoyHTTP
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20