apache/airflow

Add Prometheus Provider

Open

#20,053 opened on Dec 5, 2021

View on GitHub
 (8 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:new provider request

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Description

Prometheus supports a pushGateway option to push metrics to Prometheus. This might be useful in cases where metrics needs to be pushed in the ETL stage.

For example, BigQuery has metrics stored in information_schema tables which can be queried using Standard SQL. Its useful to run the Query and store the metrics in prometheus on a periodic basis.

https://github.com/m-lab/prometheus-bigquery-exporter

Suggestion is to add a Prometheus provider first and have a separate PR for BigQueryToPrometheus transfer feature.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Study the existing Airflow provider structure (e.g., in airflow/providers/) and implement a new Prometheus provider that integrates with Airflow's metrics system, following the patterns used by other providers.
Tech stack
pythonprometheus
Domain
observability
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonAirflow
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60