warm pools: token exchange costs 2 API calls per bootstrap, and a leader crash recreates and re-exchanges every warm worker

Open
  • Is something wrong?
#1,623 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
authentication, backend, performance

Research direction

The token exchange path is in internal/kubeexchange/kubeexchange.go (the TokenReview create near line 60 and the pod GET near line 181), called from internal/agentrpc/exchange.go, and the dispatch client's QPS and burst limits are set in main.go around lines 1692-1705. Start by reading those three spots and tracing one bootstrap to see the two API calls it makes. Done means the proposed changes are settled in the thread (caching, informer lookup, JWT reuse, separate QPS setting), with tests showing fewer API calls per exchange.

Written by the indexing model from the issue text.

Description

area/executor enhancement go priority/p3

Found in the 2026-10-09 performance and reliability re-audit (R-19, confirmed by two reviewers, inferred from cited code). Exchange transport is opt-in but required by warm pools.

Problem

Each exchange makes a TokenReview create (internal/kubeexchange/kubeexchange.go:60) and a pod GET (:181), with no cache (internal/agentrpc/exchange.go:87-109), on a client built with the dispatch limits (default 5 QPS, burst 10, main.go:1692-1705). That caps bootstraps at about 2.5 per second per replica. On a leader crash or rollout, idle warm workers exit and the reconciler recreates them: W pod creates plus at least W exchanges, about 400 s at W=1000 on defaults, while the requests queue, time out at 10 s and retry.

Proposal

  1. Cache TokenReview results by token hash for the token lifetime.
  2. Resolve the pod from the informer instead of a live GET.
  3. Skip re-exchange while the worker JWT is still valid.
  4. A dedicated QPS setting for exchange (both DEXAFLOW_* and LEOFLOW_*).
Dominant language
Go
Stars
46
Forks
1
Avg merge
10h 26m
Merged PRs (30d)
606

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dexadata/dexaflow

All issues in dexadata/dexaflow

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.