envoyproxy/envoy
Vedi su GitHubSpurious segmentation fault in flushMetricsToSinks -> MetricSnapshotImpl
Open
#10.179 aperta il 26 feb 2020
area/statsbughelp wanted
Metriche repository
- Star
- (27.997 star)
- Metriche merge PR
- (Merge medio 8g) (378 PR mergiate in 30 g)
Descrizione
Title: Spurious segmentation fault in flushMetricsToSinks -> MetricSnapshotImpl
Description: Envoy occasionally crashes when flushing metrics, which is unexpected.
Repro steps: Unsure how to reproduce. It is seen spuriously (usually <10 times per day in production fleet).
It has been observed in:
1.11.2+ https://github.com/envoyproxy/envoy/pull/9121 + internal patch1.12.2+ https://github.com/envoyproxy/envoy/pull/9121 + https://github.com/envoyproxy/envoy/pull/9509
Config:
# stats configuration
stats_flush_interval: 30s
stats_sinks:
- name: "envoy.dog_statsd"
config:
address:
socket_address:
address: "127.0.0.1"
port_value: 8200
stats_config:
stats_tags:
- tag_name: "envoy_dst_zone"
regex: 'cluster\.(?:[^.]+\.)?zone\.[-a-z0-9]+\.(([-a-z0-9]+)?\.)'
- tag_name: "envoy_src_zone"
regex: 'cluster\.(?:[^.]+\.)?zone\.(([-a-z0-9]+)?\.)'
Call Stack: From the 1.12.2 build described above:
[2020-02-26 18:12:18.680][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x0
[2020-02-26 18:12:18.681][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-02-26 18:12:18.681][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: c3c7d68a4844af3cb125330ec6527243571aa822/1.12.2.2/Clean/RELEASE/BoringSSL
[2020-02-26 18:12:18.682][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f54b8ab6390]
[2020-02-26 18:12:18.688][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Server::MetricSnapshotImpl::MetricSnapshotImpl() [0x55fe11a57415]
[2020-02-26 18:12:18.690][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::Server::InstanceUtil::flushMetricsToSinks() [0x55fe11a57b32]
[2020-02-26 18:12:18.693][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::Server::InstanceImpl::flushStatsInternal() [0x55fe11a57d7f]
[2020-02-26 18:12:18.696][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Stats::ThreadLocalStoreImpl::mergeInternal() [0x55fe11ac41b5]
[2020-02-26 18:12:18.699][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: std::_Function_handler<>::_M_invoke() [0x55fe11ac422b]
[2020-02-26 18:12:18.702][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x55fe11a9ae23]
[2020-02-26 18:12:18.705][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: event_process_active_single_queue.isra.33 [0x55fe11db34b0]
[2020-02-26 18:12:18.707][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: event_base_loop [0x55fe11db3b7f]
[2020-02-26 18:12:18.710][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: Envoy::Server::InstanceImpl::run() [0x55fe11a60eba]
[2020-02-26 18:12:18.713][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: Envoy::MainCommonBase::run() [0x55fe1144abfc]
[2020-02-26 18:12:18.716][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: main [0x55fe11412453]
[2020-02-26 18:12:18.716][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: __libc_start_main [0x7f54b86fb830]