Possible UAF when HDS cluster is removed and health check connection fails in the same I/O iteration

First Reported by: bbassingthwaite@digitalocean.com to envoy-security. @yanavlasov moved it into the open after analyzing the crash. The crash can not be triggered by untrusted traffic.

We're currently running: envoyproxy/envoy:debug-dev-583132c930fe3f8f105bbff0c0aaeae072d5d9eb (nothing significant about this version other then it was HEAD at the time) but we also know the issue exists on 1.33.0.

We currently use HDS (Health Discovery Service) to configure Envoy to health check a specific set of backends that is dynamically changing. We believe that the endpoint being health checked was configured via HDS.

This is the stack trace on the segfault we are seeing:

2025-04-15T15:39:44.009836778Z stdout F {"timestamp": "2025-04-15T15:39:44.009724Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:129", "msg": "Caught Segmentation fault, suspect faulting address 0x107940103"} 2025-04-15T15:39:44.009838128Z stdout F {"timestamp": "2025-04-15T15:39:44.009735Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:113", "msg": "Backtrace (use tools/stack_decode.py to get line numbers):"} 2025-04-15T15:39:44.009840578Z stdout F {"timestamp": "2025-04-15T15:39:44.009737Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:114", "msg": "Envoy version: https://github.com/envoyproxy/envoy-setec/commit/583132c930fe3f8f105bbff0c0aaeae072d5d9eb/1.34.0-dev/Clean/RELEASE/BoringSSL"} 2025-04-15T15:39:44.009841908Z stdout F {"timestamp": "2025-04-15T15:39:44.009739Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:116", "msg": "Address mapping: 55cbe2f21000-55cbe5bcd000 /usr/local/bin/envoy"} 2025-04-15T15:39:44.009922637Z stdout F {"timestamp": "2025-04-15T15:39:44.009899Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:123", "msg": "#0: [0x7f34122c9520]"} 2025-04-15T15:39:44.011907456Z stdout F {"timestamp": "2025-04-15T15:39:44.011871Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/1: Envoy::Upstream::HealthCheckerImplBase::ActiveHealthCheckSession::handleFailure() [0x55cbe3112682]"} 2025-04-15T15:39:44.013656617Z stdout F {"timestamp": "2025-04-15T15:39:44.013619Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/2: Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::onResetStream() [0x55cbe3104125]"} 2025-04-15T15:39:44.015322879Z stdout F {"timestamp": "2025-04-15T15:39:44.015285Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/3: Envoy::Http::Http1::ClientConnectionImpl::onResetStream() [0x55cbe509de8f]"} 2025-04-15T15:39:44.016765834Z stdout F {"timestamp": "2025-04-15T15:39:44.016728Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/4: Envoy::Http::CodecClient::onEvent() [0x55cbe4f761cf]"} 2025-04-15T15:39:44.01810871Z stdout F {"timestamp": "2025-04-15T15:39:44.018089Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/5: Envoy::Network::ConnectionImplBase::raiseConnectionEvent() [0x55cbe534390b]"} 2025-04-15T15:39:44.019462585Z stdout F {"timestamp": "2025-04-15T15:39:44.019445Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/6: Envoy::Network::ConnectionImpl::raiseEvent() [0x55cbe52d76f2]"} 2025-04-15T15:39:44.02081471Z stdout F {"timestamp": "2025-04-15T15:39:44.020797Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/7: Envoy::Network::ConnectionImpl::closeSocket() [0x55cbe52d6fd4]"} 2025-04-15T15:39:44.022214085Z stdout F {"timestamp": "2025-04-15T15:39:44.022193Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/8: Envoy::Network::ConnectionImpl::onWriteReady() [0x55cbe52dc75d]"} 2025-04-15T15:39:44.023595641Z stdout F {"timestamp": "2025-04-15T15:39:44.023578Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/9: Envoy::Network::ConnectionImpl::onFileEvent() [0x55cbe52da235]"} 2025-04-15T15:39:44.024979526Z stdout F {"timestamp": "2025-04-15T15:39:44.024962Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/10: std::__1::__function::__func<>::operator()() [0x55cbe52e4646]"} 2025-04-15T15:39:44.026369261Z stdout F {"timestamp": "2025-04-15T15:39:44.026351Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/11: std::__1::__function::__func<>::operator()() [0x55cbe52cd6b6]"} 2025-04-15T15:39:44.027734596Z stdout F {"timestamp": "2025-04-15T15:39:44.027714Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/12: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x55cbe52ce855]"} 2025-04-15T15:39:44.029098102Z stdout F {"timestamp": "2025-04-15T15:39:44.029077Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/13: event_process_active_single_queue [0x55cbe55e1af3]"} 2025-04-15T15:39:44.030431277Z stdout F {"timestamp": "2025-04-15T15:39:44.030414Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/14: event_base_loop [0x55cbe55e06e1]"} 2025-04-15T15:39:44.031747413Z stdout F {"timestamp": "2025-04-15T15:39:44.031730Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/15: Envoy::Server::InstanceBase::run() [0x55cbe4a7a8f9]"} 2025-04-15T15:39:44.033084379Z stdout F {"timestamp": "2025-04-15T15:39:44.033065Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/pull/16: Envoy::MainCommonBase::run() [0x55cbe4a1e40a]"} 2025-04-15T15:39:44.034399605Z stdout F {"timestamp": "2025-04-15T15:39:44.034382Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/17: Envoy::MainCommon::main() [0x55cbe4a1e914]"} 2025-04-15T15:39:44.035683291Z stdout F {"timestamp": "2025-04-15T15:39:44.035665Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:121", "msg": "https://github.com/envoyproxy/envoy-setec/issues/18: main [0x55cbe2f2111c]"} 2025-04-15T15:39:44.035694811Z stdout F {"timestamp": "2025-04-15T15:39:44.035680Z", "level": "critical", "scope": "backtrace", "src": "./source/server/backtrace.h:123", "msg": "https://github.com/envoyproxy/envoy-setec/pull/19: [0x7f34122b0d90]"}

This is related to a customer workload and I haven't been able to reproduce the issue. I suspect it's a race condition of an endpoint connection being reset in conjunction to the cluster or endpoint being removed via HDS. But I haven't been able to prove that.

Please let me know if you need more information.

Thanks