[DISCUSSION] Circuit breaker will remain half-open state forever when the request is blocked by upcoming rules
#1,638 opened on Jul 30, 2020
Repository metrics
- Stars
- (23,109 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
Issue Description
Type: bug report
Describe what happened (or what feature you want)
The circuit breaker won't recover from half-open state when the request is actually blocked by upcoming rules. For example, there are two degrade rules of the same resource: R1(circuit breaker state=OPEN, recoveryTimeout=10s) and R2(circuit breaker state=OPEN, recoveryTimeout=20s)
There may be circumstances when R1 has reached the recovery timepoint but R2 has not. If a request comes, there will be transformation: R1(OPEN → HALF-OPEN), R2(OPEN)
This request will be allowed by R1 but rejected by R2, thus finally blocked. The invocation won't actually occur, so it will NEVER complete. For state transformation from HALF-OPEN to OPEN/CLOSED, it should happen only when invocation completes, so for R1 the associated circuit breaker state will be HALF-OPEN forever. Actually this could happen when there are any blocked rules after R1 (not only degrade rules).
This is a fatal bug and should be carefully resolved. We may need a temporary workaround for the half-open case in 1.8.0, then improve the overall design later. Discussions are welcomed.
Original discussions can be found in https://github.com/alibaba/Sentinel/pull/1490#discussion_r462079959