Passive backend ejection (outlier detection) — not saintmode, not in VarnishCachePolicy
Maintainers usually reply within 5 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- backend, networking
Research direction
Read ghost/src/vhost_director.rs and ghost/src/backend_pool.rs to trace backend selection and reload-time backend creation, then inspect the retry work in varnish/gateway#91 because it is a prerequisite. The issue proposes per-backend failure counters and ejection timestamps, a failure signal from vcl_backend_response or vcl_backend_error, fail-open behavior, and an explicit test of interaction with keep; done means these behaviors and policy attachment are implemented and covered.
Written by the indexing model from the issue text.
Description
Passive backend ejection (outlier detection)
Follow-up to backend retry. Retry recovers the individual request; ejection stops us
from steering new requests at a backend we already know is broken.
Why not vmod_saintmode
The obvious answer is Varnish's saintmode, and it does not fit:
- Ghost owns backend resolution.
VhostDirector::resolve()returns aBackendRef
chosen byselect_backend_from_groups()inghost/src/vhost_director.rs.vmod_saintmode
works by wrapping one VCL-declared backend in a saintmode director invcl_initand
stacking those inside adirectors.round_robin. There is no VCL-visible backend to wrap:
ghost's backends are created at reload time inghost/src/backend_pool.rsand churn with
every EndpointSlice change. Saintmode assumes a static, VCL-declared topology. - It is not in the image.
docker/chaperone.Dockerfileinstalls onlyvarnish;
varnish-modulesis not present and would need a build version-matched to the bundled
Varnish. - It is a VCL-level mechanism. Exposing it through a CRD field means exposing VCL
semantics in the API, which the project avoids everywhere else.
Why not VarnishCachePolicy
VCP is a caching policy: TTL, grace, keep, cache key, bypass. Backend health is a different
concern, and folding it in turns VCP into a bag of miscellaneous Varnish knobs. The one real
overlap is keep (stale-if-error), and even that only pays off once something marks a
backend sick — which is an argument for building ejection, not for filing it under caching.
Proposed shape
A VarnishBackendPolicy (Inherited Policy, same attachment model as VCP: Gateway →
HTTPRoute → rule via sectionName), targeting Service or HTTPRoute, expressed in
Gateway-API-ish vocabulary rather than Varnish vocabulary. Envoy Gateway puts the
equivalent in BackendTrafficPolicy; there is no upstream Gateway API home for this yet.
Sketch:
spec:
targetRef: { ... }
outlierDetection:
consecutiveErrors: 5 # consecutive 5xx / connect failures before ejection
interval: 10s # evaluation window
baseEjectionTime: 30s # grows with repeat offences
maxEjectionPercent: 50 # never eject more than half a group
maxEjectionPercent matters: without it, a backend-wide outage ejects every pod and we
turn a degraded service into a hard outage.
Implementation notes
- Ghost keeps a per-backend failure counter plus an ejected-until timestamp, consulted in
select_backend_from_groups(). Ejected backends are skipped; if every backend in the
selected group is ejected, fall back to serving anyway rather than 500 (fail-open). - The failure signal needs a ghost entry point called from
vcl_backend_response/
vcl_backend_error. Same deferred-postamble constraint as the retry issue — we cannot
put flow-control returns in the preamble. - State is per-pod, not cluster-wide. Each gateway pod learns independently. That is the
same model Envoy uses and it is fine, but it should be documented. - Interaction with
keepin VarnishCachePolicy: once a backend is ejected, stale-if-error
becomes reachable, which is where the caching layer and this one meet. Worth an explicit
test. - Per-object ejection (saintmode's distinguishing feature — one bad URL does not remove the
whole backend) is deliberately out of scope for a first cut. Revisit only if there is
demand.
Prerequisite
Backend retry (varnish/gateway#91) must land first. Ejection without retry just moves the failure around.
- Dominant language
- Go
- Stars
- 16
- Forks
- 3
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 10
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from varnish/gateway
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
Maintainers usually reply within 5 days
-
beta documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
varnish/gateway#58 · 1 comment ·
Maintainers usually reply within 5 days
-
beta documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
varnish/gateway#52 · 1 comment ·
Maintainers usually reply within 5 days
-
beta enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 5 days
-
Difficulty 5/5 Over a week Newbie friendliness 12/100
varnish/gateway#123 · 1 comment ·
Maintainers usually reply within 5 days
Similar issues
-
enhancement exporter/awskinesis needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/opentelemetry-collector-contrib#53334 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
erigontech/erigon#26523 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
SocialGouv/iterion#3376 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
seaweedfs/seaweedfs#12740 · 2 comments ·
Maintainers usually reply within 1 day
-
area:indexer bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
seanb4t/codegraph-go#385 ·
Maintainers usually reply within 1 day