DFP: need per host circuit breakers (cx and pending rq)
#20,207 opened on Mar 3, 2022
Repository metrics
- Stars
- (27,997 stars)
- PR merge metrics
- (Avg merge 8d) (378 merged PRs in 30d)
Description
Today, the circuit breaker is shared across the entire DFP cluster. This is not a useful construct for something like DFP in which one actually wants per-host circuit breakers (where each synthetic host targets a DNS entry).
We need to implement per-host circuit breakers, specifically for CX and pending RQ, though it would be nice to support all of the other circuit breakers as well on a per-host basis.
A couple of options for implementing this:
- Add a code level concept of host circuit breakers, and for DFP synthetic hosts, just copy the cluster level circuit breaker settings to each host on creation. Then in the code that checks for circuit breakers, look to see if there are host level breakers, and if so, use them.
- Actually allowing specific configuration of host level breakers at the cluster level, which would make this work more generically for all cluster types.
There is some thought needed about what happens if a host overflows in the general case. Would the request just fail? Trying to pick another host based on retry settings? Etc.
@alyssawilk has offered to work on this so will defer to her for further ideas on what we should implement. Thank you Alyssa!