envoyproxy/envoy

Slow startup due DNS.

Open

#10,159 opened on Feb 25, 2020

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
area/dnshelp wantedinvestigate

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Title: One line description

Description: Envoy is slow to get ready in my IPv6 cluster. I install envoy 1.12.2 as a part of a contour environment with the attached manifest. Envoy is installed as a DaemonSet. Some of the pods can take up to 140s to get ready. I suspect some kind of race condition during startup. If I add a new initcontainer to the DaemonSet, Sleeping 5 seconds, the issue is not reproducible for me. I do also have another cluster with only IPv4 addresses, and installing with the manifest I also do not see the issue. The only things I changed compared to the original contour example is:

  • getting rid of the internal security.
  • changes in the args for envoy to be ready for v6 use contour.txt

Repro steps: kubectl apply -f contour.yaml

kubectl get pods -n projectcontour NAME READY STATUS RESTARTS AGE
contour-58f47b9d8c-4k664 1/1 Running 0 2m20s
contour-58f47b9d8c-rpkcv 1/1 Running 0 2m20s
envoy-k59m9 0/1 Running 0 2m20s
envoy-n4kw4 1/1 Running 0 2m20s
envoy-vnnq2 1/1 Running 0 2m20s
envoy-xrnr9 0/1 Running 0 2m20s

I can include the debug logs from envoy-k59m9 envoy.pod.log

envoy.json used as config. envoy.config.txt

Contributor guide

Research direction
Investigate DNS resolution timing during Envoy startup in IPv6 clusters. Check if there is a race condition when resolving the initial configuration endpoint, possibly related to IPv6 DNS query delays. Compare behavior with IPv4 and consider adding a sleep or retry mechanism.
Tech stack
cppkubernetesdocker
Domain
networkinginfrastructurecloud
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
KubernetesDNSnetworking
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30