envoyproxy/envoy

Crash in Envoy::Http::AsyncStreamImpl::closeRemote on shutdown

Open

#7.983 aberto em 20 de ago. de 2019

Ver no GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (5.373 forks)batch import
bughelp wanted

Métricas do repositório

Stars
 (27.997 stars)
Métricas de merge de PR
 (Mesclagem média 8d) (378 fundiu PRs em 30d)

Description

Seeing the following crash running off 38a4d86407668612ccbc4cba65d11be6cbd81b32:

2019-08-20_22:24:17.36421 [2019-08-20 22:24:17.364][374905][warning][main] [external/envoy/source/server/server.cc:469] caught SIGTERM
2019-08-20_22:24:17.36423 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:574] shutting down server instance
2019-08-20_22:24:17.36424 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:528] main dispatch loop exited
2019-08-20_22:24:17.36959 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation f
ault, suspect faulting address 0x0
2019-08-20_22:24:17.36962 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/
stack_decode.py to get line numbers):
2019-08-20_22:24:17.37594 [2019-08-20 22:24:17.375][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7
f1f290895d0]
2019-08-20_22:24:17.39615 [2019-08-20 22:24:17.396][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Http::AsyncStreamImpl::closeRemote() [0x8f0b46]
2019-08-20_22:24:17.40303 [2019-08-20 22:24:17.402][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Http::AsyncStreamImpl::encodeData() [0x8f5148]
2019-08-20_22:24:17.41097 [2019-08-20 22:24:17.410][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Http::Utility::sendLocalReply() [0xa05ff8]
2019-08-20_22:24:17.41454 [2019-08-20 22:24:17.414][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Http::AsyncStreamImpl::sendLocalReply() [0x8f0e69]
2019-08-20_22:24:17.41906 [2019-08-20 22:24:17.419][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Router::Filter::onUpstreamAbort() [0x8f71c3]
2019-08-20_22:24:17.42254 [2019-08-20 22:24:17.422][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Router::Filter::onUpstreamReset() [0x902bee]
2019-08-20_22:24:17.42603 [2019-08-20 22:24:17.425][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Router::Filter::UpstreamRequest::onResetStream() [0x902fb2]
2019-08-20_22:24:17.42950 [2019-08-20 22:24:17.429][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::Http1::ClientConnectionImpl::onResetStream() [0x95f30d]
2019-08-20_22:24:17.43457 [2019-08-20 22:24:17.434][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::CodecClient::onEvent() [0x8e27a4]
2019-08-20_22:24:17.44230 [2019-08-20 22:24:17.442][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Network::ConnectionImpl::raiseEvent() [0x753b25]
2019-08-20_22:24:17.44620 [2019-08-20 22:24:17.446][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Network::ConnectionImpl::closeSocket() [0x753045]
2019-08-20_22:24:17.45422 [2019-08-20 22:24:17.454][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Network::ConnectionImpl::close() [0x75489d]
2019-08-20_22:24:17.45843 [2019-08-20 22:24:17.458][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http1::ConnPoolImpl::~ConnPoolImpl() [0x88abf5]
2019-08-20_22:24:17.46249 [2019-08-20 22:24:17.462][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::Http1::ProdConnPoolImpl::~ProdConnPoolImpl() [0x88ad00]
2019-08-20_22:24:17.46651 [2019-08-20 22:24:17.466][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Upstream::ConnPoolMap<>::clearActivePools() [0x78500e]
2019-08-20_22:24:17.46989 [2019-08-20 22:24:17.469][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Upstream::ConnPoolMap<>::~ConnPoolMap() [0x78c2e5]
2019-08-20_22:24:17.47316 [2019-08-20 22:24:17.473][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x78c428]
2019-08-20_22:24:17.47640 [2019-08-20 22:24:17.476][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: std::_Hashtable<>::clear() [0x784e4a]
2019-08-20_22:24:17.47980 [2019-08-20 22:24:17.479][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::~ThreadLocalClusterManagerImpl() [0x789c2b]
2019-08-20_22:24:17.48601 [2019-08-20 22:24:17.485][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::ThreadLoc
al::InstanceImpl::shutdownThread() [0x70873a]
2019-08-20_22:24:17.48990 [2019-08-20 22:24:17.489][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Server::W
orkerImpl::threadRoutine() [0x746d49]
2019-08-20_22:24:17.49403 [2019-08-20 22:24:17.493][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Thread::T
hreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0xc879c5]
2019-08-20_22:24:17.49410 [2019-08-20 22:24:17.494][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: start_thread [0x
7f1f29081dd5]
2019-08-20_22:24:17.49412 AsyncClient 0x4981808, stream_id_: 1003858367269869698
2019-08-20_22:24:17.49413 &stream_info_:
2019-08-20_22:24:17.49413   StreamInfoImpl 0x4981960, protocol_: 1, response_code_: null, response_code_details_: upstream_reset_before_response_started{connection termination}, health_check_request_: 0, rout
e_name_:

Not 100% sure that this was introduced with our recent update, as we've been seeing crashes with similar traces on older versions.

Guia do colaborador

Direção de pesquisa
Investigue a desreferência de ponteiro nulo em AsyncStreamImpl::closeRemote durante o desligamento. Rastreie o caminho da chamada a partir do backtrace para determinar por que o objeto de stream está em um estado inconsistente quando o pool de conexões é destruído.
Pilha de tecnologia
cpp
Domain
backend
Tipo Issue
Bug
DifficultyDificuldade de implementação estimada para um novo contribuidor, de 1 para alterações muito pequenas a 5 para trabalho de nível especializado.
3
Tempo estimadoUm intervalo de tempo aproximado para um colaborador experiente investigar, implementar, testar e preparar um pull request.
Meio dia
Status da atividadeQuão disponível o issue aparece agora: novo, ativo, obsoleto, bloqueado ou aguardando entrada do mantenedor.
Ativo
ClarityCom que clareza o issue explica a mudança esperada, os critérios de aceitação e a próxima etapa.
Principalmente claro
Prerequisites
C++EnvoyHTTP
Simpatia para novatosUma pontuação de 1 a 100 que estima o quão acessível este issue é para colaboradores iniciantes.
20