No visibility on Juju controller when it gets detached from the dqlite cluster

Open
#155 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start by tracing how the charm determines and reports the controller unit's status, then find where it can check whether the Juju API is listening on port 17070. The issue gives no file or test paths, so first locate the status checks and their tests. Done means a detached controller is reported as blocked or in error so an administrator can investigate.

Written by the indexing model from the issue text.

Description

A node in a Juju cluster can get detached from the cluster (the dqlite data may be corrupted or outdated) and this can go unnoticed since Juju status will still display the unit as active and jujud is still running.

The only symptom of the issue is the fact that the Juju API in the node cannot start and the unit is effectively not listening on port 17070 which causes API calls hitting this node to fail/timeout. The affected controller Juju logs would show something like:

2026-06-05 23:44:41 ERROR juju.worker.dependency engine.go:695 "lease-manager" manifold worker returned unexpected error: catacomb 0xc0019f3a48 is dying
2026-06-05 23:44:41 DEBUG juju.worker.dependency engine.go:618 "db-accessor" manifold worker stopped: create node: failed to set bind address "@dqlite-3297041220608546238": 1
2026-06-05 23:44:41 ERROR juju.worker.dependency engine.go:695 "db-accessor" manifold worker returned unexpected error: create node: failed to set bind address "@dqlite-3297041220608546238": 1

When the issue is outdated dqlite data (there may be other I'm not aware of), a cross-inspection of /var/lib/juju/dqlite/ shows that the affected controller's dqlite data is outdated.

I'd argue that we should have better visibility on this (and potentially other issues). The easiest I can think of to achieve this is to mark the unit as blocked or in error if it's not listening on the port. This should trigger an admin to investigate and potentially resolve this and other issues.

Dominant language
Python
Stars
3
Forks
21
Avg merge
2d 22h
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from juju/juju-controller

All issues in juju/juju-controller

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.