envoyproxy/envoy

Unexpected behavior when configured with listeners that fail to bind.

Open

#3,395 opened on May 16, 2018

View on GitHub
 (10 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
bughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Description:

Envoy successfully starts when configured with listeners on ports that it fails to bind to, which results in unexpected behavior (see below).

Repro steps:

Port :10001 fail - both connections are matched to the wildcard listener 0.0.0.0:10001, even though more specific listener 127.0.0.1:10001 exists (or at least, was configured):

$ curl 192.168.9.1:10001
This server is listening on *:10001

$ curl 127.0.0.1:10001
This server is listening on *:10001

Port :10002 fail - all connections to the wildcard listener 0.0.0.0:10002 are refused, because listener 127.0.0.1:10002 was configured first:

$ curl 192.168.9.1:10002
curl: (7) Failed to connect to 192.168.9.1 port 10002: Connection refused

$ curl 127.0.0.1:10002
This server is listening on 127.0.0.1:10002

Admin and Stats Output:

$ curl -s localhost:10000/listeners
["0.0.0.0:10001","127.0.0.1:10001","127.0.0.1:10002","0.0.0.0:10002"]

$ curl -s localhost:10000/stats | grep ^listener
listener.0.0.0.0_10001.downstream_cx_active: 0
listener.0.0.0.0_10001.downstream_cx_destroy: 2
listener.0.0.0.0_10001.downstream_cx_total: 2
listener.0.0.0.0_10001.http.whatever.downstream_rq_1xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_2xx: 2
listener.0.0.0.0_10001.http.whatever.downstream_rq_3xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_4xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_5xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_1xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_2xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_3xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_4xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_5xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_1xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_2xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_3xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_4xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_5xx: 0
listener.127.0.0.1_10002.downstream_cx_active: 0
listener.127.0.0.1_10002.downstream_cx_destroy: 1
listener.127.0.0.1_10002.downstream_cx_total: 1
listener.127.0.0.1_10002.http.whatever.downstream_rq_1xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_2xx: 1
listener.127.0.0.1_10002.http.whatever.downstream_rq_3xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_4xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_5xx: 0
listener.admin.downstream_cx_active: 1
listener.admin.downstream_cx_destroy: 1
listener.admin.downstream_cx_total: 2
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 1
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener_manager.listener_added: 4
listener_manager.listener_create_failure: 2
listener_manager.listener_create_success: 16
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.total_listeners_active: 4
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
listener.0.0.0.0_10001.downstream_cx_length_ms: P0(nan,1) P25(nan,1.05) P50(nan,1.1) P75(nan,2.05) P90(nan,2.08) P95(nan,2.09) P99(nan,2.098) P99.9(nan,2.0998) P100(nan,2.1)
listener.127.0.0.1_10002.downstream_cx_length_ms: P0(nan,1) P25(nan,1.025) P50(nan,1.05) P75(nan,1.075) P90(nan,1.09) P95(nan,1.095) P99(nan,1.099) P99.9(nan,1.0999) P100(nan,1.1)
listener.admin.downstream_cx_length_ms: P0(nan,nan) P25(nan,nan) P50(nan,nan) P75(nan,nan) P90(nan,nan) P95(nan,nan) P99(nan,nan) P99.9(nan,nan) P100(nan,nan)

Config:

admin:
  access_log_path: "/dev/null"
  address:
    socket_address: { address: 127.0.0.1, port_value: 10000 }

static_resources:
  listeners:
  # :10001 - wildcard, localhost
  - name: wildcard_10001
    address:
      socket_address: { address: 0.0.0.0, port_value: 10001 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: whatever
          codec_type: AUTO
          route_config:
            virtual_hosts:
            - name: whatever
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                direct_response:
                  status: 200
                  body: { inline_string: "This server is listening on *:10001\n" }
          http_filters:
          - name: envoy.router
  - name: localhost_10001
    address:
      socket_address: { address: 127.0.0.1, port_value: 10001 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: whatever
          codec_type: AUTO
          route_config:
            virtual_hosts:
            - name: whatever
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                direct_response:
                  status: 200
                  body: { inline_string: "This server is listening on 127.0.0.1:10001\n" }
          http_filters:
          - name: envoy.router
  # :10002 - localhost, wildcard
  - name: localhost_10002
    address:
      socket_address: { address: 127.0.0.1, port_value: 10002 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: whatever
          codec_type: AUTO
          route_config:
            virtual_hosts:
            - name: whatever
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                direct_response:
                  status: 200
                  body: { inline_string: "This server is listening on 127.0.0.1:10002\n" }
          http_filters:
          - name: envoy.router
  - name: wildcard_10002
    address:
      socket_address: { address: 0.0.0.0, port_value: 10002 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: whatever
          codec_type: AUTO
          route_config:
            virtual_hosts:
            - name: whatever
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                direct_response:
                  status: 200
                  body: { inline_string: "This server is listening on *:10002\n" }
          http_filters:
          - name: envoy.router

Logs:

[info][main] source/server/server.cc:402] starting main dispatch loop
[debug][main] source/server/worker_impl.cc:91] worker entering dispatch loop
[critical][config] source/server/listener_manager_impl.cc:613] listener 'localhost_10001' failed to listen on address '127.0.0.1:10001' on worker
[debug][config] source/server/listener_manager_impl.cc:655] begin remove listener: name=localhost_10001
[debug][config] source/server/listener_manager_impl.cc:663] unknown/locked listener 'localhost_10001'. no remove
[critical][config] source/server/listener_manager_impl.cc:613] listener 'wildcard_10002' failed to listen on address '0.0.0.0:10002' on worker
[debug][config] source/server/listener_manager_impl.cc:655] begin remove listener: name=wildcard_10002
[debug][config] source/server/listener_manager_impl.cc:663] unknown/locked listener 'wildcard_10002'. no remove

Contributor guide

Research direction
Examine the listener manager code (listener manager impl.cc) to understand how it handles bind failures and why listeners are still considered active, then propose a fix that properly fails or disables listeners that fail to bind.
Tech stack
cpp
Domain
backendinfrastructure
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Envoy configurationnetwork programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40