Port-conflict fallback is persisted to k3d.yaml and never recovers, permanently moving the stack off :8080
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- docker, go, kubernetes
- Domain
- cli, devops, infrastructure
Research direction
Start in internal/stack/backend_k3d.go, especially rewriteConflictingPorts around line 289 and ensureK3dPortsAvailable around line 439; trace how the embedded template and persisted $OBOL_CONFIG_DIR/k3d.yaml are used during stack up. Reproduce the conflict and recovery sequence with the commands in the issue. Done means freed default ports are reconsidered, all four intended ingress mappings are preserved where available, and subsequent runs explain any remaining fallback.
Written by the indexing model from the issue text.
Description
Summary
A transient host-port conflict permanently degrades $OBOL_CONFIG_DIR/k3d.yaml. Once obol stack up falls back to ephemeral ingress ports, the stack never returns to 80/8080 — not on restart, not on cluster recreation — even after the conflicting process is long gone. The only recovery is obol stack init --force.
It also silently drops the stack from four ingress mappings to two.
Impact
Every user-facing instruction, doc, and the obol stack up epilogue itself says to visit http://obol.stack:8080. After this fires, that URL is wrong forever and the stack answers on a random high port instead. A user who ran two stacks once — or had anything on port 80 during a single stack up — is left permanently off the documented URL with no obvious way back, because k3d.yaml is not a file most people know to inspect.
Observed in practice: an orphaned older k3d cluster held 80/8080, so a new stack came up on 56854. After deleting the orphan and recreating the cluster, the new cluster still bound 56854 — the ports were free, but the defaults were no longer in the config to be reconsidered.
Root cause
rewriteConflictingPorts (internal/stack/backend_k3d.go:289) is one-directional. It removes conflicting mappings and appends an ephemeral fallback, but never restores a default mapping once the port frees up:
if available(c.hostPort) || owned[c.hostPort] {
hasMapping[c.containerPort] = true
continue
}
k3dConfig = strings.Replace(k3dConfig, block, "", 1) // default mapping deleted
ensureK3dPortsAvailable (internal/stack/backend_k3d.go:439) then persists the result on every stack up, and its own doc comment frames it as handling drift between init and up — but the drift only ever flows one way:
updated := stripConflictingPorts(original, u, false, "")
if updated != original {
os.WriteFile(configPath, []byte(updated), 0o600)
}
The ratchet:
stack upwith80/8080occupied → both80:80and8080:80blocks are deleted;hasMapping[80]is false, sopickPort()appends an ephemeral56854:80. Same for443. Persisted.- Ports free up. Next
stack upreads a config that now contains only56854:80. That port is available, sohasMapping[80] = trueand nothing changes. The defaults are never reconsidered — they are no longer in the file to be checked.
The 4→2 reduction falls out of the same loop: the template ships 80:80, 8080:80, 443:443, 8443:443, but the recovery path adds at most one mapping per container port.
Reproduction
# occupy the default ingress ports (e.g. a second k3d cluster, or any listener)
obol stack up # → "Default ingress ports are in use — use http://obol.stack:56854 instead"
grep -A1 "port:" $OBOL_CONFIG_DIR/k3d.yaml
# - port: 56854:80
# - port: 56855:443 # 8080:80 and 8443:443 are gone
# free the ports, then fully recreate the cluster
k3d cluster delete <the-other-cluster>
k3d cluster delete obol-stack-<id>
obol stack up
docker port k3d-obol-stack-<id>-serverlb
# 80/tcp -> 0.0.0.0:56854 # still the fallback, ports 80/8080 are free and unused
Recovery, currently the only way:
obol stack init --force # preserves the stack ID, regenerates all four mappings
obol stack up
Suggested fix
Make the rewrite reconsider defaults each run instead of mutating a shrinking config. Concretely: derive the port mappings from the embedded template on every up, apply conflict-stripping to that, and treat the on-disk k3d.yaml as derived state rather than the source of truth for ports. Then a freed port is automatically reclaimed on the next stack up.
Two smaller pieces worth fixing alongside:
- Preserve the secondary mappings — after a conflict the stack should still get both
8080-style and primary mappings where the ports allow, not collapse to one per container port. - The fallback warning is printed once, at the moment of the conflict. Since the condition is now sticky,
obol stack upshould keep surfacing why it is not on:8080on subsequent runs, and say how to reset.
Notes
--forcealready has the right concept viak3dOwnedHostPorts(a cluster's own ports are not foreign conflicts); the gap is specifically that freed foreign ports are never re-acquired.- Found while validating #802 on a live cluster; unrelated to that PR.
- Dominant language
- Go
- Stars
- 11
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ObolNetwork/obol-stack
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ObolNetwork/obol-stack#612 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
ObolNetwork/obol-stack#829 ·
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 72/100
ObolNetwork/obol-stack#828 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
ObolNetwork/obol-stack#821 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
ObolNetwork/obol-stack#807 · 1 comment ·
All issues in ObolNetwork/obol-stack
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·