gateway: certs of deleted workloads are never removed from traefik's acme.json
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- networking
Research direction
Start in pkg/gateway/gateway.go at ensureGateway and compare the corresponding startup path in pkg/gateway_light/gateway.go. Read how acme.json and the dynamic config directory are handled before Traefik starts; done means stale certificate entries are removed in both gateway variants without affecting certificates with matching configuration.
Written by the indexing model from the issue text.
Description
Note: the code lives in zosbase (
pkg/gateway), filing here as the tracking repo. Same bug affectspkg/gateway_light.
What happens
When a gateway workload is deleted, we remove its traefik routing config but leave its TLS certificate in traefik's acme.json. Traefik keeps trying to renew that certificate every day, forever.
On a long-lived public node this piles up. Real log from a mainnet gateway — 19 renewals failing in one 60-second sweep, all for domains whose deployments are long gone:
09:39:52 error msg="Error renewing certificate from LE: {test.fqdn.main.gridtesting.xyz []}"
error="acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN
looking up A for test.fqdn.main.gridtesting.xyz - check that a DNS record exists"
...
09:40:52 error msg="Error renewing certificate from LE: {lhumina.com []}"
error="acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 168.119.77.253:
Invalid response from https://lhumina.com/.well-known/acme-challenge/...: 404"
17x NXDOMAIN (domain deleted), 2x unauthorized (domain moved to another host). Several are gridtesting.xyz CI leftovers.
Why
DeleteNamedProxy — the single delete path for both name and fqdn workloads — removes the dynamic config file and the reserved-domain entry, but never touches acme.json:
pkg/gateway/gateway.go:739pkg/gateway_light/gateway.go:764(identical)
verifyDomainDestination (pkg/gateway/gateway.go:488) only runs at deploy time, so a domain that later disappears is never noticed.
Impact
Cosmetic today — live workloads renew fine, and LE's failed-validation limit is per-hostname so we aren't rate-limited. But the list only grows over a node's lifetime, and it makes gateway logs useless for spotting real problems. Every gateway node that has ever served a workload has this.
Suggested fix
Prune acme.json on gateway startup, in ensureGateway, before traefik starts: drop any entry whose domain has no matching file in the dynamic config dir.
Doing it at startup rather than in DeleteNamedProxy avoids racing traefik for the file (traefik owns acme.json while running and doesn't watch it for changes), and it self-heals nodes that have already accumulated junk. Same change needed in gateway_light.
- Dominant language
- Go
- Stars
- 95
- Forks
- 22
- Avg merge
- 2m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from threefoldtech/zos
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
threefoldtech/zos#2704 ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
threefoldtech/zos#2693 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
threefoldtech/zos#2691 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
threefoldtech/zos#2689 ·
-
threefoldtech/zos#2686 · 1 assignee ·
All issues in threefoldtech/zos
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·