envoyproxy/envoy

Improving other_host retry plugin interaction with RR lb policy

Open

#4,859 opened on Oct 25, 2018

View on GitHub
 (1 comment) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
bughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

The other_hosts retry plugin attempts to select another host N times (configurable) until it finds a host that hasn't been attempted before. There is a degenerate case here when using the RR LB:

Consider a scenario where all the endpoints in a cluster have been attempted (ie this is not the first set of retries) and still have several retry attempts remaining. Let the number of endpoints be 2, named H1 and H2, with host_selection_retry_max_attempts set to 1:

Attempt 1:
<H2 was previously selected by RR LB>
Select H1, reject since it's been attempts
Select H2, select since we have no more host selection retries
Attempt 2:
<H2 was previously selected by RR LB>
Select H1, reject since it's been attempts
Select H2, select since we have no more host host selection retries

Essentially, H2 will be continued to be selected because the host selection retries iterate over all the endpoints in the cluster. The issue is present when using RR and when (host_selection_retry_max_attempts + 1) % number of endpoints = 0 and when no other requests are being sent to the cluster on the same worker (doing so would affect the hosts selected from the LB, breaking the pattern).

To fix this, adding an option to reset the list of attempted hosts when the LB was unable to find a suitable host would allow using RR without fear of it getting "stuck" like this targeting a single host.

Contributor guide