Support HTTPRoute retry (no backend retry path exists today)

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes

Research direction

Start with the existing route cache policy plumbing from routing.json through ghost.json, then read ghost/src/vhost_director.rs and internal/vcl/postamble.vcl. Decide how retry settings trigger Varnish retries, avoid the failed backend, handle backoff, and treat non-idempotent requests; check the Gateway API experimental-field requirements. Done means retry behavior is covered by ghost-side VTC tests and an end-to-end test with a failing pod.

Written by the indexing model from the issue text.

Description

enhancement experimental

Backend retry: no retry path exists today

Field: HTTPRoute.spec.rules[].retry (HTTPRouteRetry) — Extended support, experimental channel only.

Problem

There is no backend retry anywhere in the gateway. We deliberately do not generate
vcl_backend_error (see AGENTS.md: it is reserved for user VCL), builtin VCL delivers
the 503, and nothing ever calls return(retry).

Consequence: a pod that is still in the EndpointSlice but failing to accept connections
returns 503 straight to the client, with no attempt at a sibling pod in the same backend
group. Kubernetes readiness probes eventually evict it, but that window is seconds to tens
of seconds, and for its duration a fixed fraction of requests to that route simply fail —
1/N of them per retry-free attempt, where N is the pod count in the selected group.

This is the single largest availability gap in the data path right now.

Proposed scope

Implement HTTPRouteRetry:

  • attempts — max retries for one backend request (maps onto bereq.retries /
    max_retries)
  • codes — response status codes that trigger a retry
  • backoff — minimum delay between attempts (GEP-2257 duration)

Per the API docs, implementations SHOULD retry connection errors (disconnect, reset,
timeout, TCP failure) whenever a retry stanza is configured, regardless of codes.

Implementation notes
  • Plumbing: retry config is per-rule, so it rides the existing path —
    routing.json → ghost.json → ghost route entry. Same shape as the cache policy
    fields already attached to routes.
  • Triggering the retry: return(retry) must NOT go in the preamble's
    vcl_backend_response, for the same reason return(pass) does not go in the preamble's
    vcl_recv — it terminates VCL and user VCL never runs. Use the established deferred
    pattern: ghost sets a header (e.g. X-Ghost-Retry), and a postamble
    vcl_backend_response checks it and calls return(retry) after user VCL has run.
    This means adding a vcl_backend_response to internal/vcl/postamble.vcl, which
    currently only carries vcl_recv.
  • Avoiding the same pod: on retry, Varnish re-enters vcl_backend_fetch and calls
    the director again. select_backend_from_groups() in ghost/src/vhost_director.rs is
    pure weighted random, so a retry can land on the backend that just failed. Ghost should
    read bereq.retries and exclude the previously-selected backend for this fetch.
    Cheapest version: stash the last-tried backend key on bereq and filter it out of the
    candidate list. This is the prerequisite for varnish/gateway#92 and makes retry actually
    useful rather than statistically pointless.
  • Backoff: Varnish has no native inter-retry delay. May need vmod_std's sleep
    equivalent or be documented as unsupported initially — worth deciding explicitly rather
    than silently ignoring the field.
  • Idempotency: the Gateway API retry spec does not restrict retries by method.
    Decide whether to retry non-idempotent requests and document the choice.
Caveats
  • retry is <gateway:experimental>, so it is absent from standard-install.yaml, which
    is what the docs tell users to install. Supporting it means either documenting the
    experimental channel as a prerequisite for this feature, or gating the controller on
    the field's presence.
  • There is no SupportHTTPRouteRetry conformance feature and no retry conformance tests
    in gateway-api v1.5.0, so this needs our own test coverage (VTC for ghost-side backend
    exclusion, plus an e2e with a deliberately broken pod).
Dominant language
Go
Stars
16
Forks
3
Avg merge
5d 15h
Merged PRs (30d)
10

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from varnish/gateway

All issues in varnish/gateway

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.