envoyproxy/envoy

503 + NR due to the inconsistency between the order in which the configuration takes effect and the order in xDS push

Open

#13,910 opened on Nov 5, 2020

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (27,997 stars) (5,373 forks)batch import
area/xdsbughelp wanted

Description

We currently use a control plane software to dynamically create clusters and routes. However, in practice, we have found that if we create a route immediately after creating a cluster. For a very small period of time, requests to access the route return 503 + NR.

We speculate that the process of cluster warming causes the cluster configuration to take effect later than the route configuration takes effect. But not for sure.

I wonder if the community has encountered a similar problem? Any suggestions?

Contributor guide

Tech stack
cpp
Domain
backendinfrastructure
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Understanding of Envoy proxy internalsKnowledge of xDS protocolFamiliarity with cluster warming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
Investigate the cluster warming mechanism in Envoy's source code (likely in source/common/upstream/) and the order of configuration application in xDS handling (source/common/config/). Review the xDS push order for clusters and routes to identify the race condition. Check if there are any existing comments or linked issues about similar problems. The issue reports a 503 + NR error when creating a route immediately after a cluster, suggesting the cluster may not be ready. Focus on the warming logic and ensure routes are not applied before clusters are fully configured.