feat(operator): run the read plane on every replica and leader-elect the write plane

Closed
  • Is something wrong?
#320 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes, rust
Domain
distributed-systems, infrastructure

Research direction

Start by reading charts/grid-operator/values.yaml and the operator main loop under operator/src; the issue identifies the existing replica constraint and the missing Lease coordination. Trace where scraping, the signals listener, reconcilers, and the SWIM advertiser start, then inspect the chart’s deployment settings. Done means read-plane work runs on every replica, write-plane work is Lease-gated, and the chart supports multiple replicas with a disruption budget of one.

Written by the indexing model from the issue text.

Description

triage/needs-triage

The operator runs as one replica and has no leader election: charts/grid-operator/values.yaml says replicas must stay at 1 until multi-replica operation exists, and nothing in operator/src takes a coordination Lease. That makes it the single point on the routing path. When the operator pod restarts, the gateway's pollers get nothing new, every provider's samples age out of the gateway's load window, every site reads as unmeasured, and grid_site_route degrades to its capacity arm and never sheds. It fails open, but it routes blind until the operator is back.

The two halves of the operator want different answers.

The read plane, scraping each provider's EPP and serving /v1/site/signals, is idempotent and can run on every replica with no leader. The gateway and the peers poll a Service, so any replica answers. Two replicas scrape every provider, which doubles a load the EPP already refreshes every few milliseconds, and hold equivalent state for the same reason two gateway replicas do: same source, offset by scrape phase.

The write plane, reconciling the CRDs, writing the serving ConfigMap and status, and advertising one SWIM member per site, must run on one replica at a time, on the holder of a coordination.k8s.io Lease, the standard controller pattern.

Proposed: split the main loop so the scrapers and the signals listener start unconditionally, gate the reconcilers and the SWIM advertiser on a Lease, and lift the replicas constraint in the chart with a disruption budget of one. The gateway side of the same story, seeding learned draw state from a sibling replica at startup, is separate and does not depend on this.

Dominant language
Rust
Stars
10
Forks
24
Avg merge
1d 11h
Merged PRs (30d)
136

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from praxis-proxy/grid

All issues in praxis-proxy/grid

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.