envoyproxy/envoy

Resource-based circuit breaking

Open

#3,332 opened on May 9, 2018

View on GitHub
 (4 comments) (5 reactions) (1 assignee)C++ (5,373 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Description

Envoy should have the ability to circuit break on system resources like CPU.

Circuit breakers at ingress are used to protect our hosts from resource exhaustion. To determine circuit breaker thresholds, we run a "redline" test, which increasingly ramps traffic on a single host until it degrades. We note rq_active, then set the threshold less some buffer.

Over time this ends up being a poor approximation for the real bottleneck for most of our services, CPU:

  • If any of the service's dependencies slow down, a single host can handle additional concurrency without exhausting its local resources.
  • If a service (the local service or a downstream) ships code that changes the overall load profile and/or the overall request mix, the circuit breaker may no longer protect the service or circuit break early.

Working out the platform-dependent implementation and the algorithm will be the fun part. I'd like to get a first impression from other users before getting into that.

Contributor guide

Research direction
Investigate the existing circuit breaker implementation in Envoy, understand the current threshold based model, and design a CPU based resource detection mechanism. Consider platform specific APIs (e.g., /proc/stat on Linux) and propose an algorithm to dynamically adjust circuit breaking thresholds based on CPU usage.
Tech stack
None
Domain
infrastructure
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++distributed systemscircuit breakers
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20