apache/airflow
View on GitHubProvide configuration option to configure path of OTEL metrics exporter
Open
#42,492 opened on Sep 26, 2024
good first issuekind:featuretelemetry
Repository metrics
- Stars
- (44,809 stars)
- PR merge metrics
- (Avg merge 7d 18h) (834 merged PRs in 30d)
Description
Description
Currently Airflow does not allow to configure path in OTEL metrics exporter
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")
resource = Resource(attributes={SERVICE_NAME: "Airflow"})
protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}/v1/metrics"
And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path
/opentelemetry/api/v1/push/insert/0/opentelemetry/api/v1/push
It will be great to have the metrics path to be configurable, like the following
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex: "/opentelemetry/api/v1/push"
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")
resource = Resource(attributes={SERVICE_NAME: "Airflow"})
protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}{path}"
Use case/motivation
Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct