Passive backend ejection (outlier detection) — not saintmode, not in VarnishCachePolicy

Open
#92 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes

Research direction

Read ghost/src/vhost_director.rs and ghost/src/backend_pool.rs to trace backend selection and reload-time backend creation, then inspect the retry work in varnish/gateway#91 because it is a prerequisite. The issue proposes per-backend failure counters and ejection timestamps, a failure signal from vcl_backend_response or vcl_backend_error, fail-open behavior, and an explicit test of interaction with keep; done means these behaviors and policy attachment are implemented and covered.

Written by the indexing model from the issue text.

Description

enhancement

Passive backend ejection (outlier detection)

Follow-up to backend retry. Retry recovers the individual request; ejection stops us
from steering new requests at a backend we already know is broken.

Why not vmod_saintmode

The obvious answer is Varnish's saintmode, and it does not fit:

  • Ghost owns backend resolution. VhostDirector::resolve() returns a BackendRef
    chosen by select_backend_from_groups() in ghost/src/vhost_director.rs. vmod_saintmode
    works by wrapping one VCL-declared backend in a saintmode director in vcl_init and
    stacking those inside a directors.round_robin. There is no VCL-visible backend to wrap:
    ghost's backends are created at reload time in ghost/src/backend_pool.rs and churn with
    every EndpointSlice change. Saintmode assumes a static, VCL-declared topology.
  • It is not in the image. docker/chaperone.Dockerfile installs only varnish;
    varnish-modules is not present and would need a build version-matched to the bundled
    Varnish.
  • It is a VCL-level mechanism. Exposing it through a CRD field means exposing VCL
    semantics in the API, which the project avoids everywhere else.
Why not VarnishCachePolicy

VCP is a caching policy: TTL, grace, keep, cache key, bypass. Backend health is a different
concern, and folding it in turns VCP into a bag of miscellaneous Varnish knobs. The one real
overlap is keep (stale-if-error), and even that only pays off once something marks a
backend sick — which is an argument for building ejection, not for filing it under caching.

Proposed shape

A VarnishBackendPolicy (Inherited Policy, same attachment model as VCP: Gateway →
HTTPRoute → rule via sectionName), targeting Service or HTTPRoute, expressed in
Gateway-API-ish vocabulary rather than Varnish vocabulary. Envoy Gateway puts the
equivalent in BackendTrafficPolicy; there is no upstream Gateway API home for this yet.

Sketch:

spec:
  targetRef: { ... }
  outlierDetection:
    consecutiveErrors: 5        # consecutive 5xx / connect failures before ejection
    interval: 10s               # evaluation window
    baseEjectionTime: 30s       # grows with repeat offences
    maxEjectionPercent: 50      # never eject more than half a group

maxEjectionPercent matters: without it, a backend-wide outage ejects every pod and we
turn a degraded service into a hard outage.

Implementation notes
  • Ghost keeps a per-backend failure counter plus an ejected-until timestamp, consulted in
    select_backend_from_groups(). Ejected backends are skipped; if every backend in the
    selected group is ejected, fall back to serving anyway rather than 500 (fail-open).
  • The failure signal needs a ghost entry point called from vcl_backend_response /
    vcl_backend_error. Same deferred-postamble constraint as the retry issue — we cannot
    put flow-control returns in the preamble.
  • State is per-pod, not cluster-wide. Each gateway pod learns independently. That is the
    same model Envoy uses and it is fine, but it should be documented.
  • Interaction with keep in VarnishCachePolicy: once a backend is ejected, stale-if-error
    becomes reachable, which is where the caching layer and this one meet. Worth an explicit
    test.
  • Per-object ejection (saintmode's distinguishing feature — one bad URL does not remove the
    whole backend) is deliberately out of scope for a first cut. Revisit only if there is
    demand.
Prerequisite

Backend retry (varnish/gateway#91) must land first. Ejection without retry just moves the failure around.

Dominant language
Go
Stars
16
Forks
3
Avg merge
5d 15h
Merged PRs (30d)
10

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from varnish/gateway

All issues in varnish/gateway

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.