No visibility on Juju controller when it gets detached from the dqlite cluster
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- backend, observability
Research direction
Start by tracing how the charm determines and reports the controller unit's status, then find where it can check whether the Juju API is listening on port 17070. The issue gives no file or test paths, so first locate the status checks and their tests. Done means a detached controller is reported as blocked or in error so an administrator can investigate.
Written by the indexing model from the issue text.
Description
A node in a Juju cluster can get detached from the cluster (the dqlite data may be corrupted or outdated) and this can go unnoticed since Juju status will still display the unit as active and jujud is still running.
The only symptom of the issue is the fact that the Juju API in the node cannot start and the unit is effectively not listening on port 17070 which causes API calls hitting this node to fail/timeout. The affected controller Juju logs would show something like:
2026-06-05 23:44:41 ERROR juju.worker.dependency engine.go:695 "lease-manager" manifold worker returned unexpected error: catacomb 0xc0019f3a48 is dying
2026-06-05 23:44:41 DEBUG juju.worker.dependency engine.go:618 "db-accessor" manifold worker stopped: create node: failed to set bind address "@dqlite-3297041220608546238": 1
2026-06-05 23:44:41 ERROR juju.worker.dependency engine.go:695 "db-accessor" manifold worker returned unexpected error: create node: failed to set bind address "@dqlite-3297041220608546238": 1
When the issue is outdated dqlite data (there may be other I'm not aware of), a cross-inspection of /var/lib/juju/dqlite/ shows that the affected controller's dqlite data is outdated.
I'd argue that we should have better visibility on this (and potentially other issues). The easiest I can think of to achieve this is to mark the unit as blocked or in error if it's not listening on the port. This should trigger an admin to investigate and potentially resolve this and other issues.
- Dominant language
- Python
- Stars
- 3
- Forks
- 21
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from juju/juju-controller
-
metrics-endpoint-relation-created hook fails with ValueError on k8s when bind_address returns service FQDNMay be free again @gfouillet claimed this 89 days ago, and no pull request is open. Open
juju/juju-controller#130 · 1 assignee ·
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
juju/juju-controller#112 · 10 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 40/100
juju/juju-controller#96 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
juju/juju-controller#93 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
juju/juju-controller#92 · 1 comment ·
All issues in juju/juju-controller
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
snap-stanford/Biomni#380 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/playwright-python#3361 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
cubrid-lab/sqlalchemy-cubrid#1397 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
cubrid-lab/pycubrid#1307 ·
Maintainers usually reply within 1 day
-
[Bug]: Installer hides SSL certificate errors as bare URLError, and the suggested fix can't workOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ShaftHQ/chaosengine-harness#143 ·
Maintainers usually reply within 1 day