Grant server autoconfigure fails permanently when two controller pods race on create

Open
#2,636 3 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 4 days

@AryanP123 is already working on this.

Since Nov 18, 2026.

  • #2638 by @AryanP123 — open

Assessment

This issue has not been assessed yet.

Description

Describe the bug
With -grant-server-autoconfigure, startup does a plain get-then-create for Certificate skupper-grant-server-ca and SecuredAccess skupper-grant-server. If two controller pods start at once, both can see NotFound and Create; the loser gets AlreadyExists, logs once, and never retries.

That pod also never watches the grant-server SecuredAccess, so it never learns the URL. If it survives (e.g. during a rollout), AccessGrants stay Pending with empty status.url until the controller is restarted.

Auto configuration of grant server failed ... certificates.skupper.io "skupper-grant-server-ca" already exists

How To Reproduce
On a fresh install with the 2.3.0 cluster-scope YAML:

  1. Reset grant-server resources, then start the controller and immediately change the Deployment so two pods overlap:
kubectl -n skupper scale deploy/skupper-controller --replicas=1
kubectl -n skupper set env deploy/skupper-controller FOO=bar
  1. Or scale to 2 replicas from a clean state.

Surviving pod may log the AlreadyExists error. Create an AccessGrant and watch status.url stay empty while SecuredAccess is Ready.

Expected behavior
Autoconfigure should tolerate concurrent create (adopt/reconcile) and retry on failure, so grant URLs still get set without a restart.

Environment details

  • Skupper CLI: N/A
  • Skupper Operator (if applicable): controller 2.3.0 / main
  • Platform: kubernetes (also affects OpenShift; same code path)

Additional context
Found this while building a multi-cluster demo of skupper with llm-d and openshift AI.

Dominant language
Go
Stars
702
Forks
93
Avg merge
4d 5h
Merged PRs (30d)
8

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from skupperproject/skupper

All issues in skupperproject/skupper

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.