metrics-endpoint-relation-created hook fails with ValueError on k8s when bind_address returns service FQDN

Open
#130 0 comments 0 reactions 1 assignee View on GitHub

@gfouillet is already working on this.

Since Aug 24, 2026.

Assessment

This issue has not been assessed yet.

Description

Symptom

On a k8s controller, integrating juju-controller with prometheus-k8s via
controller:metrics-endpoint causes the metrics-endpoint-relation-created
hook to fail repeatedly with:

Traceback (most recent call last):
  File ".../charms/prometheus_k8s/v0/prometheus_scrape.py", line 1523, in set_scrape_job_spec
    self._set_unit_ip()
  File ".../prometheus_scrape.py", line 1554, in _set_unit_ip
    unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)
  File ".../ops/model.py", line 1084, in _network_get
    return Network(self._backend.network_get(name, relation_id))
  File ".../ops/model.py", line 1230, in NetworkInterface.__init__
    subnet = ipaddress.ip_network(address)
  File "/usr/lib/python3.12/ipaddress.py", line 83, in ip_network
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: 'controller-0.controller-service-endpoints.controller-ctrl-<id>.svc.cluster.local'
          does not appear to be an IPv4 or IPv6 network

Followed immediately by:

WARNING unit.controller/0.juju-log metrics-endpoint:1:
  0 containers are present in metadata.yaml and refresh_event was not specified.
  Defaulting to update_status. Metrics IP may not be set in a timely fashion.

The uniter retries every ~30 s for ~13 minutes; SAAS offer stays in error;
check_prometheus_targets exhausts its 30-retry budget.

Reproduction (CI gating)

https://jenkins.juju.canonical.com/job/test-controllercharm-test-prometheus-microk8s/,
builds #3703 #3706 #3707 #3708 #3724 #3725 (all 4.1-beta2.x / main /
feature/controller-snap):

juju offer controller.controller:metrics-endpoint
juju deploy prometheus-k8s --channel 1/stable --trust
juju relate prometheus-k8s admin/controller.controller
# controller SAAS → 'error' after ~5s
# after 12m50s:  check_prometheus_targets prometheus-k8s 0 failed after 30 retries

Charm revs observed failing: juju-controller-157 (4.1-beta2 main),
juju-controller-328 (4.0.15.x). 3.6.28 / juju-controller-361 does not
exhibit the failure.

Tracked from juju side at https://warthogs.atlassian.net/browse/JUJU-10266.

Root cause

lib/charms/prometheus_k8s/v0/prometheus_scrape.py:_set_unit_ip calls

unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)

binding.network.bind_address accesses ops.model.NetworkInterface.__init__,
which calls ipaddress.ip_network(address). When the controller pod has just
been created on k8s, network_get returns the service-endpoints DNS FQDN
(<unit>.controller-service-endpoints.<ns>.svc.cluster.local, matching the
ControllerServiceFQDNTemplate in juju/internal/provider/kubernetes/constants/constants.go),
which is not a valid CIDR. The ValueError propagates as an uncaught
exception, the hook fails, and because the charm has no containers: and no
refresh_event, the IP is not re-evaluated — every ~30 s retry repeats
the same race.

Note: the same _set_unit_ip line exists in the upstream main at
revision 58 of prometheus_k8s/v0/prometheus_scrape.py, with the same
missing guard.

Suggested fixes

Either of:

  1. Charmlib — wrap the bind_address access in a try/except and defer
    the relation-data write via a deferred event handler when the binding is
    unresolved; or generally wrap set_scrape_job_spec so that _set_unit_ip
    failures do not fail the hook.

  2. Charm — add a containers: stanza with refresh_event: <container>-pebble-ready
    in metadata.yaml and observe PebbleReady for metrics-endpoint. This
    also suppresses the "0 containers in metadata.yaml" warning and gives
    the IP a retry path through the operator framework.

Local mitigation until either lands

In juju/tests/suites/controllercharm/prometheus.sh, after juju relate,
add a wait_for "controller SAAS status == active" before
check_prometheus_targets, so failures surface with the actual error rather
than a 12m50s retry timeout (see https://warthogs.atlassian.net/browse/JUJU-10266).

Likely related
  • #93 "Wrong host address for metrics scrape job" — same _set_unit_ip
    line (main @ line 1692 / rev-157 @ line 1554) returns a non-IP target
    without crashing. I believe #93 and this issue share a root cause:
    missing guard on bind_address and no retry path when the binding is
    not yet IP-resolvable.
  • #112 "metrics-endpoint-relation-created on every unit fail in cluster
    mode" — different failure (APIError: user "juju-metrics-r1" already exists) on the same hook, but a third race condition on the relation
    lifecycle.
Dominant language
Python
Stars
3
Forks
21
Avg merge
2d 22h
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from juju/juju-controller

All issues in juju/juju-controller

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.