[Bug]: ate-api-server rejects every token when the cluster's service account issuer is not kubernetes.default.svc

Open
  • Is something wrong?
#3,326 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
authentication, backend

Research direction

Start in cmd/ate-setup/internal/steps/create.go at buildAuthenticationConfig and trace how the service account issuer determines the authentication configuration. Check how the cluster's discovery document, service account CA, and token are available, then verify token authentication for a non-default issuer and that discovery failures are logged with their cause.

Written by the indexing model from the issue text.

Description

area/api-machinery area/dev-infra area/identity kind/bug
What happened?

On Talos the service account issuer is https://<control-plane-ip>:6443. buildAuthenticationConfig (cmd/ate-setup/internal/steps/create.go) adds certificateAuthorityFile and discoveryTokenFile only when the issuer equals https://kubernetes.default.svc (or .cluster.local). For any other issuer, ate-api-server tries unauthenticated discovery against the API server with the system roots, and every client call fails with Unauthenticated: invalid bearer token. Nothing in the server log names the cause.

Expected Behavior

ate-api-server verifies tokens for any issuer that the cluster's own discovery document returns, or logs why discovery failed.

Steps to Reproduce

on a cluster whose service account issuer is the API server address (Talos default)
run ate-setup deploy ate-system
then any kubectl-ate call
it fails with invalid bearer token.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

v0.6.0 (d3f2363); the same code is on main as of 2026-10-27

Kubernetes Version & Environment

Kubernetes 1.37.1, Talos 1.14.4, Cilium

Host OS & Architecture

Talos Linux, amd64

Relevant Logs and Diagnostic Output
Additional Context

Suggested fix: add the pod's service account CA and token whenever the issuer is the cluster's own (or offer a setting for them), and log the discovery error. We worked around it by patching the generated authentication config. Related: #3271. Found while evaluating Substrate v0.6.0 on Talos 1.14 / Kubernetes 1.37 / Cilium; background: https://groups.google.com/g/ate-dev/c/WeQaFvHYZb4

Confirmation
  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.
Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.