Implement publishable P2P (per-pod L4 NLB + ExternalAddress)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- aws, go, kubernetes, terraform
- Domain
- backend, devops, infrastructure, networking
Research direction
Start with docs/designs/sei-publishable-p2p-nlb.md, then trace the PR sequence through api/v1alpha1, internal/controller/seinodedeployment, internal/controller/node, and the referenced planner entry points. Run the relevant envtests as each slice is implemented; done requires the Service, ExternalAddress, peer resolution, rollout configuration, and external TCP connectivity to work together.
Written by the indexing model from the issue text.
Description
Problem
Sei pods today are reachable for HTTP (RPC/EVM/REST) via the existing Istio L7 path → shared NLB → private pod, but they are not publicly dialable as P2P peers over TCP/26656. This blocks validators (in a deterministic set without sentries) and advertised full nodes from being addressable by ecosystem peers, and prevents internal SNDs from peering each other across clusters.
The design (platform/docs/designs/sei-publishable-p2p-nlb.md, merged in sei-protocol/platform#714) closes this gap with an opt-in networking.tcp: {} sub-struct: when set, the SeiNode reconciler stamps a per-pod Service type=LoadBalancer (AWS NLB, target-type: ip, scheme=internet-facing, cross-zone=true), reads Service.status.loadBalancer.ingress[0].hostname, and writes SeiNode.Status.ExternalAddress = "<hostname>:26656". The planner already reads ExternalAddress into p2p.external_address at planner.go:697; this issue makes that pipeline reachable.
Impact
Unblocks:
- Validator P2P publishability — deterministic set peers each other over stable NLB DNS endpoints; combined with
unconditional_peer_idscross-config (ops, not this issue) gives stable cross-cluster validator mesh. - Cross-SND peer discovery — new
PeerSource.seinodeDeploymentvariant resolves sibling SND children'sStatus.ExternalAddressintopersistent_peers, replacing the EC2-tag-only discovery for K8s-resident publishers. - Ecosystem-parity for advertised full nodes — partner peers can list us as a persistent peer.
Harbor stays excluded until the Cilium overlay is replaced (CGNAT pod IPs aren't VPC-routable → target-type: ip fails); controller fails closed via a VPC-CIDR routability check.
Proposed approach
Six independently-shippable PRs. Sequence dependencies: PR-1 → PR-2 → PR-3 (the chain that makes the field gateable and propagated). PR-4 lands in parallel with PR-3 (independent code path; needs PR-1's types only). PR-5 lands any time before PR-6. PR-6 waits on PR-3 + PR-4 + PR-5.
PR-1 — Types layer (contract-only, no behavior change)
api/v1alpha1/networking_types.go: addHTTP *HTTPConfig+TCP *TCPConfigtoNetworkingConfig; addHTTPEnabled()accessor (legacynetworking: {}→ treated ashttp: {}for backcompat).api/v1alpha1/seinode_types.go: addNetworking *NetworkingConfigtoSeiNodeSpec; flipExternalAddressdoc-comment from "SeiNodeDeployment controller" → "SeiNode controller".api/v1alpha1/common_types.go: addSeiNodeDeploymentPeerSourcetoPeerSourceunion; bump XValidation CEL sum.- CRD regen + smoke test: existing manifests (Networking-nil and legacy
networking: {}) still validate.
PR-2 — SND template propagation
internal/controller/seinodedeployment/: propagateSpec.Template.Networkingintochild.Spec.Networkingat child-creation time (single line in the existing template-merge path).- Envtest: SND with
networking.tcp: {}in template → child SeiNodes carrySpec.Networking.TCP != nil.
PR-3 — LB Service + ExternalAddress + Service-ready gate (the heavy PR)
internal/controller/node/external_address.go(NEW):- Stamp
Service type=LoadBalancer(<seinode>-p2p) whenSpec.Networking.TCP != niland pod IP is insideSEI_VPC_CIDRenv (VPC-CIDR routability gate). - Annotations:
aws-load-balancer-type=nlb,scheme=internet-facing,nlb-target-type=ip,cross-zone-load-balancing-enabled=true;externalTrafficPolicy: Local; onlyTCP/26656. - Watch Service status; on
loadBalancer.ingress[0].hostnamepopulated → writeStatus.ExternalAddress = "<host>:26656"(bare host:port, no nodeId@). - Empty-host guard:
if newHost == "" { return nil }— don't clear on transient absence. - Service-ready gate: when
tcp != nil, hold STS creation untilStatus.ExternalAddressis written. (Without this, the first sidecar render has emptyp2p.external_address; planner doesn't detect runtime override drift atplanner.go:706.) - Opt-out cleanup: when
Spec.Networking.TCP == nil, delete the Service inline.
- Stamp
internal/controller/node/controller.go: confirm existingOwns(&corev1.Service{})at:204covers the new Service (predicate audit); add RBAC verbs if missing (create/update/delete on services).- Manager env:
SEI_VPC_CIDRplumbed via the controller Deployment (set inplatform/clusters/prod/sei-k8s-controller/manager-patch.yaml— same pattern asSEI_GATEWAY_PUBLIC_DOMAIN). - Envtest: bootstrap with
tcp: {}→ STS doesn't appear until Service has a (fake) hostname; opt-out path deletes the Service.
PR-4 — seinodeDeployment peer-source variant
internal/controller/node/peers.go: extendreconcilePeersswitch with the new variant; addresolveSNDPeersthat lists matching SNDs → lists child SeiNodes → emits each child'sStatus.ExternalAddressashost:portintoResolvedPeers. The sidecar's existingCollectAndSetPeerstask atinternal/planner/group.go:59prependsnodeID@at task-build time (:26657/statusquery) — same path thelabelvariant feeds. Not a new sidecar capability.internal/controller/node/controller.go:Watches(&SeiNodeDeployment{}, mapperFn); naive mapper (re-reconcile all SeiNodes in namespace on any SND change) is acceptable at the ~20-publishable scale; tighten to selector-matching scope when fanout becomes a concern.- RBAC: cluster-scoped
seinodedeploymentsget/list/watch. - Envtest: consumer SND with
peers: [{ seinodeDeployment: ... }]→ resolves to producer SND children'sExternalAddress.
PR-5 — Node SG opening (in sei-protocol/platform)
- Terraform:
tcp/26656 from 0.0.0.0/0ingress on the Sei-node SG. Lands before the first publishable SND.
PR-6 — First publishable rollout (in sei-protocol/platform)
- Enable
networking.tcp: {}on one advertised full-node SND. - Verify: external
nc <nlb-host> 26656;:26657/net_infoshows DNS form inlisten_addr; a remote node connects with<node_id>@<host>:26656. - Validator rollout follows the same shape, one at a time across the deterministic set, with
unconditional_peer_idscross-config landed first (runbook).
Out of scope
- Config-hash STS-template annotation for automatic runtime
tcp: {}add — v2. Runtime adds require a manual pod delete afterStatus.ExternalAddresspopulates (NLB hostnames don't change once allocated, so this doesn't recur). - EIP-attached NLBs (stable IP, not just DNS) — DNS form only.
- NLB-attached SG (defense in depth) — relies on node-SG.
- Dedicated publishable EC2NodeClass / scoped SG — opens 26656 on shared node SG.
- Orphan-cleanup walker for SND scale-down events — opt-out (
tcp: {}→ nil) cleanup IS in PR-3; scale-down (replica drop) orphans wait for v2. - Harbor publishability — gated on Cilium overlay → VPC-routable CNI migration. Controller fails closed.
- Validator
unconditional_peer_idscross-config — operational runbook, not a controller change. - CometBFT sentry architecture — explicitly not adopted; deterministic validator set publishes directly.
Relevant experts
- kubernetes-specialist — PR-2, PR-3, PR-4 (controller-runtime, CRD types, envtest)
- platform-engineer — PR-3 manager env wiring, PR-5 (Terraform), PR-6 (manifest)
- sei-network-specialist — PR-6 verification (CometBFT NodeInfo + dial semantics)
References
- Design:
sei-protocol/platform,docs/designs/sei-publishable-p2p-nlb.md(merged in #714) - Planner integration point:
internal/planner/planner.go:697(commonOverridesalready readsStatus.ExternalAddress) - Existing Service watch:
internal/controller/node/controller.go:204 - Drift-detection limit acknowledged:
internal/planner/planner.go:706(buildRunningPlanonly checks image + sidecar reapproval; ExternalAddress changes after Running don't propagate) - AWS LBC annotation precedent: existing Istio gateway in
platform/clusters/prod/gateway/gateway.yaml
One-way doors (already approved in design)
Status.ExternalAddressformat:"<host>:26656"(bare host:port; host is raw NLB*.elb.<region>.amazonaws.com, no vanity domain)NetworkingConfig.HTTP/TCPsub-struct field names (presence-signals)- Legacy
networking: {}interpreted asnetworking: { http: {} } SeiNodeSpec.Networkingfield add (additive; propagated from SND template)PeerSource.seinodeDeploymentvariant (additive; bump union CEL)- Keep
peersfield name (not renamed)
- Dominant language
- Go
- Stars
- 1
- Forks
- 2
- Avg merge
- 2h 29m
- Merged PRs (30d)
- 56
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from sei-protocol/sei-k8s-controller
-
Difficulty 5/5 Over a week Newbie friendliness 32/100
-
Difficulty 5/5 Over a week Newbie friendliness 32/100
sei-protocol/sei-k8s-controller#457 · 2 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
All issues in sei-protocol/sei-k8s-controller
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·