Swarm Task SD container labels missing

What did you do?

I am trying to access container labels on tasks returned by the dockerswarm_sd tasks role. In particular, the images I am using are all built with an org.opencontainers.image.version label that I would like to surface in my prometheus metrics. This label is visible when I inspect the containers running these images and I can filter and format these labels when inspecting containers.

What did you expect to see?

According to the prometheus configuration documentation I linked above; I should be able to see a __meta_dockerswarm_container_label_<labelname> label for each of the labels on the container.

What did you see instead? Under which circumstances?

I was not able to see any container labels at all. I was able to see every other task __meta label listed in the documentation, but there are no __meta_dockerswarm_container_label_<labelname> labels available on any discovered tasks despite the fact that the containers definitely have labels. I was able to see __meta_dockerswarm_service_label_ values, but none for the containers even though the docs say I should see container labels too.

Looking a little further, it seems the tasks docker API does not return container labels, despite having a section in the output named Labels this section does not appear to contain any labels at all on any of my tasks, it is always empty even though the task I am inspecting definitely has container labels, image labels, and service labels, none of them appear when inspecting a task. Inspecting the container, image, or service reveals the labels but when inspecting tasks the label section is always empty. Does prometheus rely solely on the tasks api during service discovery (I'm guessing not since I can see service labels at least)? Do I need to do something to tell docker swarm to return labels when inspecting a task? Did this feature work before on an older version of docker? Maybe its a bug with docker?

System information

Linux 5.15.0-69-generic x86_64

Prometheus version

prometheus, version 2.43.0 (branch: HEAD, revision: edfc3bcd025dd6fe296c167a14a216cab1e552ee)
  build user:       root@8a0ee342e522
  build date:       20230321-12:56:07
  go version:       go1.19.7
  platform:         linux/amd64
  tags:             netgo,builtinassets

Prometheus configuration file

The relevant job config below. It might also be useful to know I am using the docker socket proxy project to connect prometheus: https://github.com/Tecnativa/docker-socket-proxy


  - job_name: swamp
    dockerswarm_sd_configs:
      - host: tcp://{{ docker_proxy_service_name }}:2375
        role: tasks
    relabel_configs:
      # Only keep containers that should be running.
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep
      # Only keep containers that have a `prometheus-port` label.
      - source_labels: [__meta_dockerswarm_service_label_prometheus_port]
        regex: .+
        action: keep
      # get the port from this target
      - source_labels: [__address__]
        regex: '[^\s]+:(\d+)'
        target_label: __tmp_found_port
      # keep only if this is the port we're looking for
      - source_labels: [__meta_dockerswarm_service_label_prometheus_port]
        target_label: __tmp_found_port
        action: keepequal
      # limit to one network
      - source_labels: [__meta_dockerswarm_service_label_prometheus_network]
        target_label: __meta_dockerswarm_network_name
        action: keepequal
      # label with the node (better than slot)
      - source_labels: [__meta_dockerswarm_node_hostname]
        target_label: docker_node
      # track the node availability
      - source_labels: [__meta_dockerswarm_node_availability]
        target_label: node_availability
      # track the version
      - source_labels: [__meta_dockerswarm_container_label_org_opencontainers_image_version]
        target_label: version
      # track the image creation time
      - source_labels: [__meta_dockerswarm_container_label_org_opencontainers_image_created]
        target_label: image_build_date
      # replace address with prometheus-replacement-port if available
      - source_labels: [__address__, __meta_dockerswarm_service_label_prometheus_replace_port]
        separator: ':'
        regex: '([^:]+):[^:]+:(\d+)'
        replacement: "${1}:${2}"
        target_label: __address__
      # override address
      - source_labels: [__meta_dockerswarm_service_label_prometheus_address]
        regex: '([^\s]+)'
        target_label: __address__
      # override scheme
      - source_labels: [__meta_dockerswarm_service_label_prometheus_scheme]
        regex: '([^\s]+)'
        target_label: __scheme__
      - source_labels: [__meta_dockerswarm_service_name]
        target_label: instance

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

```text
No relevant logs, prometheus and docker socket proxy both show no errors and all 200 response codes.