elastic/beats

docker autodiscover hints & networks

Open

#8498 opened on Sep 29, 2018

View on GitHub
 (9 comments) (3 reactions) (1 assignee)Go (9,173 stars) (3,451 forks)batch import
Team:Cloudnative-MonitoringTeam:Integrationscontainersenhancementgood first issue

Description

Hi,

I'm configuring my servers monitoring with metricbeat, and I have an issue: I'm using docker autodiscover hints to find containers to monitor, eg:


  traefik:
    image: traefik:1.6.6
    # some config unrelated to this issue skipped
    deploy:
      mode: global
    labels:
      co.elastic.metrics/module: traefik
      co.elastic.metrics/metricsets: health
      co.elastic.metrics/hosts: '$${data.host}:8080'
      co.elastic.metrics/period: 10s
    networks:
      - web
      - default
      - metrics

networks:
  default:
  web:
    name: web
    driver: overlay
    attachable: true
  metrics:
    name: metrics
    external: true

here metrics is the network metricbeat is attached to, the traefik containers has multiple IPs for the multiple networks it's attached to, so ${data.host} get one of the IP (first? random?) but it's not the correct one. that means metricbeat is not able to query the traefik API, and I have this message instead of the metrics in the metricbeat index:

"failed to sample health: error making http request: Get http://10.0.24.31:8080/health: dial tcp 10.0.24.31:8080: connect: network is unreachable"

10.0.24 is the IP of traefik but for another container

traefik itself deals with this problem when using autodiscover by adding a label to specify wich network to use (in the exemple I have above, I use the "web" network to let traefik communicate with other containers so I add the label traefik.docker.network: web)

I think a similar label should be added to metricbeat, here I would like to add co.elastic.metrics/network: metrics

an alternative would be to add all IPs inside the autodiscover event data, so that I could use this label instead for the hosts: co.elastic.metrics/hosts: '${data.networks.metrics.ip}:8080'

Contributor guide

Tech stack
godocker
Domain
observabilitydevops
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Docker networkingMetricbeat autodiscoveryGo programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Examine the metricbeat autodiscover logic in the libbeat/autodiscover directory. Look at how docker labels are parsed and how container IPs are resolved. The user suggests adding a new label 'co.elastic.metrics/network' or exposing all network IPs. Investigate the current behavior in docker/autodiscover.go and consider implementing the proposed solution. Review the comments for additional context on the desired implementation.