apache/airflow

Provide configuration option to configure path of OTEL metrics exporter

オープン

#42,492 opened on 2024/09/26

 (2 件のコメント) (7 件のリアクション) (0 人の担当者)Python (16,781 件のフォーク)batch import
good first issuekind:featuretelemetry

Repository metrics

Stars
 (44,809 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Description

Currently Airflow does not allow to configure path in OTEL metrics exporter

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}/v1/metrics"

And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path

  • /opentelemetry/api/v1/push
  • /insert/0/opentelemetry/api/v1/push

It will be great to have the metrics path to be configurable, like the following

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex:  "/opentelemetry/api/v1/push"
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}{path}"

Use case/motivation

Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

コントリビューターガイド