apache/airflow

Provide configuration option to configure path of OTEL metrics exporter

Open

#42 492 ouverte le 26 sept. 2024

Voir sur GitHub
 (2 commentaires) (7 réactions) (0 assignés)Python (16 781 forks)batch import
good first issuekind:featuretelemetry

Métriques du dépôt

Stars
 (44 809 stars)
Métriques de merge PR
 (Merge moyen 7j 18h) (834 PRs mergées en 30 j)

Description

Description

Currently Airflow does not allow to configure path in OTEL metrics exporter

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}/v1/metrics"

And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path

  • /opentelemetry/api/v1/push
  • /insert/0/opentelemetry/api/v1/push

It will be great to have the metrics path to be configurable, like the following

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex:  "/opentelemetry/api/v1/push"
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}{path}"

Use case/motivation

Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Guide contributeur