envoyproxy/envoy

Spurious segmentation fault in flushMetricsToSinks -> MetricSnapshotImpl

Open

#10.179 aberto em 26 de fev. de 2020

Ver no GitHub
 (6 comments) (0 reactions) (0 assignees)C++ (5.373 forks)batch import
area/statsbughelp wanted

Métricas do repositório

Stars
 (27.997 stars)
Métricas de merge de PR
 (Mesclagem média 8d) (378 fundiu PRs em 30d)

Description

Title: Spurious segmentation fault in flushMetricsToSinks -> MetricSnapshotImpl

Description: Envoy occasionally crashes when flushing metrics, which is unexpected.

Repro steps: Unsure how to reproduce. It is seen spuriously (usually <10 times per day in production fleet).

It has been observed in:

Config:

# stats configuration

stats_flush_interval: 30s
stats_sinks:
  - name: "envoy.dog_statsd"
    config:
      address:
        socket_address:
          address: "127.0.0.1"
          port_value: 8200
stats_config:
  stats_tags:
    - tag_name: "envoy_dst_zone"
      regex: 'cluster\.(?:[^.]+\.)?zone\.[-a-z0-9]+\.(([-a-z0-9]+)?\.)'
    - tag_name: "envoy_src_zone"
      regex: 'cluster\.(?:[^.]+\.)?zone\.(([-a-z0-9]+)?\.)'

Call Stack: From the 1.12.2 build described above:

[2020-02-26 18:12:18.680][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x0
[2020-02-26 18:12:18.681][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-02-26 18:12:18.681][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: c3c7d68a4844af3cb125330ec6527243571aa822/1.12.2.2/Clean/RELEASE/BoringSSL
[2020-02-26 18:12:18.682][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f54b8ab6390]
[2020-02-26 18:12:18.688][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Server::MetricSnapshotImpl::MetricSnapshotImpl() [0x55fe11a57415]
[2020-02-26 18:12:18.690][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::Server::InstanceUtil::flushMetricsToSinks() [0x55fe11a57b32]
[2020-02-26 18:12:18.693][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::Server::InstanceImpl::flushStatsInternal() [0x55fe11a57d7f]
[2020-02-26 18:12:18.696][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Stats::ThreadLocalStoreImpl::mergeInternal() [0x55fe11ac41b5]
[2020-02-26 18:12:18.699][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: std::_Function_handler<>::_M_invoke() [0x55fe11ac422b]
[2020-02-26 18:12:18.702][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x55fe11a9ae23]
[2020-02-26 18:12:18.705][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: event_process_active_single_queue.isra.33 [0x55fe11db34b0]
[2020-02-26 18:12:18.707][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: event_base_loop [0x55fe11db3b7f]
[2020-02-26 18:12:18.710][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: Envoy::Server::InstanceImpl::run() [0x55fe11a60eba]
[2020-02-26 18:12:18.713][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: Envoy::MainCommonBase::run() [0x55fe1144abfc]
[2020-02-26 18:12:18.716][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: main [0x55fe11412453]
[2020-02-26 18:12:18.716][2563][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: __libc_start_main [0x7f54b86fb830]

Guia do colaborador