apache/airflow

Provide configuration option to configure path of OTEL metrics exporter

Open

#42.492 aperta il 26 set 2024

Vedi su GitHub
 (2 commenti) (7 reazioni) (0 assegnatari)Python (16.781 fork)batch import
good first issuekind:featuretelemetry

Metriche repository

Star
 (44.809 star)
Metriche merge PR
 (Merge medio 7g 18h) (834 PR mergiate in 30 g)

Descrizione

Description

Currently Airflow does not allow to configure path in OTEL metrics exporter

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}/v1/metrics"

And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path

  • /opentelemetry/api/v1/push
  • /insert/0/opentelemetry/api/v1/push

It will be great to have the metrics path to be configurable, like the following

def get_otel_logger(cls) -> SafeOtelLogger:
    host = conf.get("metrics", "otel_host")  # ex: "breeze-otel-collector"
    port = conf.getint("metrics", "otel_port")  # ex: 4318
    path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex:  "/opentelemetry/api/v1/push"
    prefix = conf.get("metrics", "otel_prefix")  # ex: "airflow"
    ssl_active = conf.getboolean("metrics", "otel_ssl_active")
    # PeriodicExportingMetricReader will default to an interval of 60000 millis.
    interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None)  # ex: 30000
    debug = conf.getboolean("metrics", "otel_debugging_on")

    resource = Resource(attributes={SERVICE_NAME: "Airflow"})

    protocol = "https" if ssl_active else "http"
    endpoint = f"{protocol}://{host}:{port}{path}"

Use case/motivation

Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Guida contributor