envoyproxy/envoy

Hot restarter kill sidecar immediately when current Envoy process is in INITIALIZING state

Open

#4,797 opened on Oct 19, 2018

View on GitHub
 (2 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
bughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Bug Template

Title: Hot restarter kill sidecar immediately when current Envoy process is in INITIALIZING state

Description:

When hot_restarter.py receives HUP while the current envoy process is still having flags = INITIALIZING, hot_restarter.py simply exit with ABORT error (this is a silent death with nothing printed to stderr). This would kill Envoy sidecar container immediately. It would be ideal to provide an option telling hot_restarter.py to keep current Envoy process under this situation.

flags checking: https://github.com/envoyproxy/envoy/blob/master/source/server/hot_restart_impl.cc#L102

Repro steps:

Configure Envoy to a unreachable xDS server.
Use hot_restarter.py to start Envoy within a container. Then docker kill <container ID> --signal HUP

Config:

Static clusters + EDS.

static_resources:
  listeners:
  - name: listener_testdb
    address:
      socket_address: { address: 0.0.0.0, port_value: 53306 }
    filter_chains:
    - filters:
      - name: envoy.tcp_proxy
        config:
          stat_prefix: testdb_egress
          cluster: test_db_cluster
  clusters:
  - name: test_db_cluster
    connect_timeout: 1s
    type: EDS
    lb_policy: ROUND_ROBIN
    eds_cluster_config:
      eds_config:
        api_config_source:
          api_type: GRPC
          grpc_services:
            envoy_grpc:
              cluster_name: control_tower
  - name: control_tower
    connect_timeout: 1s
    type: STRICT_DNS
    http2_protocol_options: {}
    load_assignment:
      cluster_name: control_tower
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: xds.sample.com
                port_value: 10080

Call Stack: I did not bother to build a debug binary. Instead, I used gdb to step through

Contributor guide