gateway: certs of deleted workloads are never removed from traefik's acme.json

Open
#2,705 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
networking

Research direction

Start in pkg/gateway/gateway.go at ensureGateway and compare the corresponding startup path in pkg/gateway_light/gateway.go. Read how acme.json and the dynamic config directory are handled before Traefik starts; done means stale certificate entries are removed in both gateway variants without affecting certificates with matching configuration.

Written by the indexing model from the issue text.

Description

Note: the code lives in zosbase (pkg/gateway), filing here as the tracking repo. Same bug affects pkg/gateway_light.

What happens

When a gateway workload is deleted, we remove its traefik routing config but leave its TLS certificate in traefik's acme.json. Traefik keeps trying to renew that certificate every day, forever.

On a long-lived public node this piles up. Real log from a mainnet gateway — 19 renewals failing in one 60-second sweep, all for domains whose deployments are long gone:

09:39:52 error msg="Error renewing certificate from LE: {test.fqdn.main.gridtesting.xyz []}"
  error="acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN
  looking up A for test.fqdn.main.gridtesting.xyz - check that a DNS record exists"
...
09:40:52 error msg="Error renewing certificate from LE: {lhumina.com []}"
  error="acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 168.119.77.253:
  Invalid response from https://lhumina.com/.well-known/acme-challenge/...: 404"

17x NXDOMAIN (domain deleted), 2x unauthorized (domain moved to another host). Several are gridtesting.xyz CI leftovers.

Why

DeleteNamedProxy — the single delete path for both name and fqdn workloads — removes the dynamic config file and the reserved-domain entry, but never touches acme.json:

  • pkg/gateway/gateway.go:739
  • pkg/gateway_light/gateway.go:764 (identical)

verifyDomainDestination (pkg/gateway/gateway.go:488) only runs at deploy time, so a domain that later disappears is never noticed.

Impact

Cosmetic today — live workloads renew fine, and LE's failed-validation limit is per-hostname so we aren't rate-limited. But the list only grows over a node's lifetime, and it makes gateway logs useless for spotting real problems. Every gateway node that has ever served a workload has this.

Suggested fix

Prune acme.json on gateway startup, in ensureGateway, before traefik starts: drop any entry whose domain has no matching file in the dynamic config dir.

Doing it at startup rather than in DeleteNamedProxy avoids racing traefik for the file (traefik owns acme.json while running and doesn't watch it for changes), and it self-heals nodes that have already accumulated junk. Same change needed in gateway_light.

Dominant language
Go
Stars
95
Forks
22
Avg merge
2m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from threefoldtech/zos

All issues in threefoldtech/zos

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.