provider runner swallows the manager error, leaving the controller alive without a health endpoint after losing the lease

Open Beginner friendly
  • Is something wrong?
#10,064 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

  • #10106 by @omkar619-dev — closed without merging

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
86/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
infrastructure

Research direction

Start with internal/provider/runner/runner.go:72-76 and compare its error handling with internal/gatewayapi/runner/runner.go:304. Trace internal/cmd/server.go:100-115 and internal/provider/kubernetes/kubernetes.go:419-433 to verify that a lost lease reaches the main error path; done means the process exits and exposes the lease error instead of waiting for the liveness probe.

Written by the indexing model from the issue text.

Description

triage

Description:

When the leader lease is lost, the manager shuts down but the process stays up. It has no
provider left, /healthz on :8081 is gone with the manager, and nothing terminates it. We saw
this twice in one day on two separate clusters. Each time the standby replica took over within a
few seconds, and each time the old pod hung around until the kubelet liveness probe killed it
40-45 seconds later.

The process survives because of internal/provider/runner/runner.go:72-76:

go func() {
    if err := p.Start(ctx); err != nil {
        r.Logger.Error(err, "unable to start provider")
    }
}()

Provider.Start does return the manager's error (internal/provider/kubernetes/kubernetes.go:419-433),
but it is only logged here. It never reaches RunnerErrors, and the main loop in
internal/cmd/server.go:100-115 only returns on l.Errors() or ctx.Done(), so the process keeps
going. internal/gatewayapi/runner/runner.go:304 does it the other way and stores the error.

I would expect losing the lease to end the process, so the pod restarts on its own exit and the
reason is visible. Right now the only thing that recovers it is the liveness probe, and the
resulting event says nothing about the lease. Without a probe configured, the pod would sit there
indefinitely.

Repro steps:

Any failure to renew the lease triggers it, the cause of the renewal failure does not matter. In
our two cases the client-side rate limiter held the renewal request until the deadline passed,
and in the other the lease GET timed out. From the logs below the sequence is:

  1. Run two replicas with leader election enabled, defaults otherwise.
  2. Make the leader fail to renew its lease.
  3. The standby acquires the lease and takes over.
  4. The former leader keeps running with /healthz closed, until something external kills it.

Environment:

Envoy Gateway v1.8.3, Kubernetes provider on EKS 1.35, two controller replicas, leader election
at its defaults (leaseDurationSeconds: 15, no client.rateLimit or leaderElection overrides
in the EnvoyGateway config). Data plane is envoy v1.38.4, not involved. The code path is
unchanged on main (3ec9a01).

Logs:

08:20:49.782 provider  Failed to update lease  lock=envoy-gateway-system/5b9825d2.gateway.envoyproxy.io
                       error="client rate limiter Wait returned an error: context deadline exceeded"
08:20:49.782 provider  Failed to renew lease
08:20:49.783 provider  unable to start provider  error="leader election lost"   (runner/runner.go:74)
08:20:49.783 provider  Stopping and waiting for non leader election runnables
08:20:49.783 provider  shutting down server  name="health probe"  addr="[::]:8081"
08:20:55.523           the standby replica acquires the lease
08:21:34     kubelet   Liveness probe failed: connection refused on :8081
             kubelet   Container envoy-gateway failed liveness probe, will be restarted
08:21:34.781           "shutting down"  (cmd/server.go:112, the ctx.Done() branch, i.e. SIGTERM)

The other occurrence looked the same, only with Error retrieving lease lock instead of the rate
limiter message.

Dominant language
Go
Stars
3.1k
Forks
940
Avg merge
2d 8h
Merged PRs (30d)
177

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from envoyproxy/gateway

All issues in envoyproxy/gateway

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.