apache/airflow

Provide configuration option to configure path of OTEL metrics exporter

Open

#42,492 创建于 2024年9月26日

在 GitHub 查看
 (2 评论) (7 反应) (0 负责人)Python (16,781 fork)batch import
good first issuekind:featuretelemetry

仓库指标

Star
 (44,809 star)
PR 合并指标
 (平均合并 7天 18小时) (30 天内合并 834 个 PR)

描述

Description

Currently Airflow does not allow to configure path in OTEL metrics exporter

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}/v1/metrics"

And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path

  • /opentelemetry/api/v1/push
  • /insert/0/opentelemetry/api/v1/push

It will be great to have the metrics path to be configurable, like the following

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex:  "/opentelemetry/api/v1/push"
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}{path}"

Use case/motivation

Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

贡献者指南