Grant server autoconfigure fails permanently when two controller pods race on create
Maintainers usually reply within 4 days
Assessment
This issue has not been assessed yet.
Description
Describe the bug
With -grant-server-autoconfigure, startup does a plain get-then-create for Certificate skupper-grant-server-ca and SecuredAccess skupper-grant-server. If two controller pods start at once, both can see NotFound and Create; the loser gets AlreadyExists, logs once, and never retries.
That pod also never watches the grant-server SecuredAccess, so it never learns the URL. If it survives (e.g. during a rollout), AccessGrants stay Pending with empty status.url until the controller is restarted.
Auto configuration of grant server failed ... certificates.skupper.io "skupper-grant-server-ca" already exists
How To Reproduce
On a fresh install with the 2.3.0 cluster-scope YAML:
- Reset grant-server resources, then start the controller and immediately change the Deployment so two pods overlap:
kubectl -n skupper scale deploy/skupper-controller --replicas=1
kubectl -n skupper set env deploy/skupper-controller FOO=bar
- Or scale to 2 replicas from a clean state.
Surviving pod may log the AlreadyExists error. Create an AccessGrant and watch status.url stay empty while SecuredAccess is Ready.
Expected behavior
Autoconfigure should tolerate concurrent create (adopt/reconcile) and retry on failure, so grant URLs still get set without a restart.
Environment details
- Skupper CLI: N/A
- Skupper Operator (if applicable): controller 2.3.0 / main
- Platform: kubernetes (also affects OpenShift; same code path)
Additional context
Found this while building a multi-cluster demo of skupper with llm-d and openshift AI.
- Dominant language
- Go
- Stars
- 702
- Forks
- 93
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 8
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from skupperproject/skupper
-
Controller Deployment discovery rejects shorter pod-template hashesPossibly taken @c-kruse claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
skupperproject/skupper#2641 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
skupperproject/skupper#2596 ·
Maintainers usually reply within 4 days
-
Controller drops informer deletion tombstonesPossibly taken @c-kruse claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
skupperproject/skupper#2639 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 48/100
skupperproject/skupper#2635 ·
Maintainers usually reply within 4 days
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
skupperproject/skupper#2632 ·
Maintainers usually reply within 4 days
All issues in skupperproject/skupper
Similar issues
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
bug ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
RedHatInsights/insights-ingress-go#946 · 4 comments ·
Maintainers usually reply within 1 day