diagnose state reconcile: rebuild state.yaml from the live cluster
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- infrastructure
Research direction
Start in internal/reality beside the persisted-keyed RefreshState, then read #1187 for consensus managed-state readback, #1225 for drift, and #1233 for overwrite policy. The issue also names the diagnostics framework (#1111) and per-component checks (#1114) as dependencies. Done means read-only reconcile reports stale, orphaned, and drifted entries, while gated --fix rebuilds state from discoverable live fields, reports unreconstructable fields, and passes the stated acceptance checks.
Written by the indexing model from the issue text.
Description
Part of the diagnostics epic (#1110). Depends on the diagnostics framework (#1111) and the per-component checks (#1114).
The recovery path for a state.yaml that is missing, corrupt, or diverged from the live cluster. This is the one mutating command in the diagnostics surface — everything else is read-only.
Why
Today there is no way to rebuild state.yaml from reality:
consensusChecker.RefreshStateiterates the persistedConsensusNodesmap and keys each lookup off the persisted identity (OrbitName/NodeId/Namespace). With state missing or an entry absent, it returns nothing — it enriches known entries, it does not discover them.consensus node installguards on the live capsule and blocks (PrecheckConsensusNotInstalled, unless--force), and the post-run flush then writes an incomplete state — so "delete state and re-install" makes things worse, not better.
A hand-edit of state.yaml is the only recovery today. This command replaces that.
Scope
- Cluster discovery (new reality capability): list the live managed resources for a component (e.g. every
ConsensusCapsule+ itsOrbit) without relying on persisted identity, and reconstruct the corresponding state entries from the live CRs (identity + the live-readable fields the checker already knows how to read). - Reconcile report (read-only): show the three-way divergence — entries in state but not in the cluster (stale), entries in the cluster but not in state (orphans — today's exact failure), and entries present on both with field drift (via
internal/driftproducers). - Repair (mutating):
diagnose state reconcile [--fix](ordiagnose state --fix) rewritesstate.yamlfrom the reconstructed live view. Gated by the block-unless-override policy (#1233): interactive runs require an explicit override before overwriting; the daemon reconciles but records what it overwrote (audit trail). Writes go through the normal atomic write + hashing.
solo-provisioner diagnose state reconcile [--component <c>] [--namespace <ns>] # report divergence (read-only)
solo-provisioner diagnose state reconcile --fix [--force] # rebuild state.yaml from the cluster (gated)
Acceptance
- On a host with a deployed consensus node but no
state.yaml, the read-only reconcile lists the node as an orphan (in cluster, not in state) with an actionable hint. --fixreconstructs a validstate.yaml(identity + live-readable fields) from the live cluster; a subsequentconsensus node diagnosereports no divergence.- Interactive
--fixthat would overwrite diverged state blocks without an override and proceeds with it (#1233); the daemon path proceeds and logs the overwrite. - Fields the live cluster cannot report (e.g.
ManagedSpecwhere it is not fully readable back) are reported as un-reconstructable rather than silently guessed.
Notes
- Builds on #1187 (consensus managed-state readback) and #1225 (drift). Discovery is the inverse of the persisted-keyed
RefreshStateand belongs beside it ininternal/reality. - Supersedes the epic's "
doctor --fixtracked separately" note for the state-reconstruction case; health-remediation--fixremains a separate future.
- Dominant language
- Go
- Stars
- 3
- Forks
- 1
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 16
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from hashgraph/solo-weaver
-
ci: pin Go version in Build and Unit Test workflows (>=1.26.6 floats to 1.27.2 and breaks golangci-lint)Possibly taken @leninmehedy claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
hashgraph/solo-weaver#1255 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
hashgraph/solo-weaver#1248 ·
Maintainers usually reply within 1 day
-
Design
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
hashgraph/solo-weaver#1118 ·
Maintainers usually reply within 1 day
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
hashgraph/solo-weaver#879 ·
Maintainers usually reply within 1 day
-
Improvement Story
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
hashgraph/solo-weaver#864 ·
Maintainers usually reply within 1 day
All issues in hashgraph/solo-weaver
Similar issues
-
unbounded value for n in `/v1/logs` can panic/OOMPossibly taken A pull request linked to this issue is open or already merged. Opensmall item
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
canonical/pebble#1127 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 89/100
open-policy-agent/opa#9366 · 1 reaction ·
Maintainers usually reply within 1 day
-
defect
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
nats-io/nats-server#8752 ·
Maintainers usually reply within 1 day
-
enhancement needs-acceptance wg/router-models-inference-runtime
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vllm-project/semantic-router#5015 · 1 comment ·
Maintainers usually reply within 1 day
-
fileserver: precompressed sidecar's Content-Encoding and Content-Length end up on error responsesPossibly taken @NitinKumar004 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
caddyserver/caddy#8204 ·
Maintainers usually reply within 1 day