Envoy 1.17 crashing with validation context secret discovery in ads mode on start up

Title: Envoy 1.17 crashing with validation context secret discovery in ads mode on start up

Description:

Envoy is crashing on start up when a cluster is configured with transport socket with UpstreamTlsContext with validation context secret discovery in ads mode. Would expect it not to crash. Also not a security issue according to security group.

Repro steps:

Run envoy 1.17 with this config, or any config that has validation_context_sds_secret_config in ads mode.

Admin and Stats Output:

No output because envoy won't start

Config: envoy-crash.txt

Logs:

Include the access logs and the Envoy logs.

Call Stack:

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:323] initializing epoch 0 (base id=0, hot restart version=11.104)

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:325] statically linked extensions:

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.udp_listeners: quiche_quic_listener, raw_udp_listener

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.dubbo_proxy.protocols: dubbo

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.resolvers: envoy.ip

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.compression.decompressor: envoy.compression.gzip.decompressor

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.health_checkers: envoy.health_checkers.redis

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.dubbo_proxy.serializers: dubbo.hessian2

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.dubbo_proxy.route_matchers: default

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.quic_client_codec: quiche

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.bootstrap: envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.http.cache: envoy.extensions.http.cache.simple

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.retry_priorities: envoy.retry_priorities.previous_priorities

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.thrift_proxy.transports: auto, framed, header, unframed

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, starttls, tls

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.upstreams: envoy.filters.connection_pools.tcp.generic

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.compression.compressor: envoy.compression.gzip.compressor

[2021-03-16 09:10:24.008][62050][info][main] [external/envoy/source/server/server.cc:327] envoy.quic_server_codec: quiche

[2021-03-16 09:10:24.023][62050][info][main] [external/envoy/source/server/server.cc:343] HTTP header map info:

[2021-03-16 09:10:24.025][62050][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size

[2021-03-16 09:10:24.025][62050][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size

[2021-03-16 09:10:24.025][62050][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size

[2021-03-16 09:10:24.026][62050][warning][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size

[2021-03-16 09:10:24.026][62050][info][main] [external/envoy/source/server/server.cc:346] request header map: 616 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id

[2021-03-16 09:10:24.026][62050][info][main] [external/envoy/source/server/server.cc:346] request trailer map: 128 bytes:

[2021-03-16 09:10:24.026][62050][info][main] [external/envoy/source/server/server.cc:346] response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id

[2021-03-16 09:10:24.026][62050][info][main] [external/envoy/source/server/server.cc:346] response trailer map: 152 bytes: grpc-message,grpc-status

[2021-03-16 09:10:24.029][62050][info][main] [external/envoy/source/server/server.cc:486] admin address: 0.0.0.0:2001

[2021-03-16 09:10:24.031][62050][info][main] [external/envoy/source/server/server.cc:634] runtime: layers:

• name: base

  static_layer:

  {}

• name: admin

  admin_layer:

  {}

[2021-03-16 09:10:24.031][62050][info][config] [external/envoy/source/server/configuration_impl.cc:125] loading tracing configuration

[2021-03-16 09:10:24.031][62050][info][config] [external/envoy/source/server/configuration_impl.cc:85] loading 0 static secret(s)

[2021-03-16 09:10:24.031][62050][info][config] [external/envoy/source/server/configuration_impl.cc:91] loading 2 cluster(s)

[2021-03-16 09:10:24.036][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0

[2021-03-16 09:10:24.036][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):

[2021-03-16 09:10:24.036][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 5c801b25cae04f06bf48248c90e87d623d7a6283/1.17.0/clean-getenvoy-be6132a-envoy/RELEASE/BoringSSL

[2021-03-16 09:10:24.037][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7f0290f35630]

[2021-03-16 09:10:24.056][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: Envoy::Secret::SdsApi::initialize() [0x55af1cae5ffe]

[2021-03-16 09:10:24.075][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: std::__1::__call_once() [0x55af1d77dd58]

[2021-03-16 09:10:24.091][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::Init::TargetHandleImpl::initialize() [0x55af1cf2842a]

[2021-03-16 09:10:24.108][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::Init::ManagerImpl::initialize() [0x55af1ced5b7a]

[2021-03-16 09:10:24.125][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::Upstream::ClusterImplBase::onPreInitComplete() [0x55af1ccb8df8]

[2021-03-16 09:10:24.141][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: std::__1::__function::__func<>::operator()() [0x55af1ccdf699]

[2021-03-16 09:10:24.158][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Network::DnsResolverImpl::PendingResolution::onAresGetAddrInfoCallback() [0x55af1cac7f8b]

[2021-03-16 09:10:24.175][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: ares_getaddrinfo [0x55af1cb7a763]

[2021-03-16 09:10:24.192][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Network::DnsResolverImpl::resolve() [0x55af1cac9668]

[2021-03-16 09:10:24.208][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: Envoy::Upstream::StrictDnsClusterImpl::ResolveTarget::startResolve() [0x55af1ccdda41]

[2021-03-16 09:10:24.225][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: Envoy::Upstream::StrictDnsClusterImpl::startPreInit() [0x55af1ccdd869]

[2021-03-16 09:10:24.242][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Upstream::ClusterImplBase::initialize() [0x55af1ccb8b65]

[2021-03-16 09:10:24.258][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Upstream::ClusterManagerInitHelper::addCluster() [0x55af1cb00c17]

[2021-03-16 09:10:24.275][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: Envoy::Upstream::ClusterManagerImpl::ClusterManagerImpl() [0x55af1cb041c4]

[2021-03-16 09:10:24.292][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: Envoy::Upstream::ProdClusterManagerFactory::clusterManagerFromProto() [0x55af1cb1069d]

[2021-03-16 09:10:24.309][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Server::Configuration::MainImpl::initialize() [0x55af1cd9cf61]

[2021-03-16 09:10:24.325][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: Envoy::Server::InstanceImpl::initialize() [0x55af1ca9bbce]

[2021-03-16 09:10:24.342][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: Envoy::Server::InstanceImpl::InstanceImpl() [0x55af1ca95f07]

[2021-03-16 09:10:24.359][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: std::__1::make_unique<>() [0x55af1b21f254]

[2021-03-16 09:10:24.375][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: Envoy::MainCommonBase::MainCommonBase() [0x55af1b21e626]

[2021-03-16 09:10:24.392][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: Envoy::MainCommon::MainCommon() [0x55af1b21f9d4]

[2021-03-16 09:10:24.408][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: Envoy::MainCommon::main() [0x55af1b21fdb3]

[2021-03-16 09:10:24.425][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: main [0x55af1b21e13c]

[2021-03-16 09:10:24.425][62050][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: __libc_start_main [0x7f0290b7a555]

Segmentation fault (core dumped)