apache/airflow

Add SRV DNS record support to http connections

Open

#9,186 opened on Jun 8, 2020

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
good first issuekind:featureprovider:http

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Description

SRV records are DNS records that resolve to a hostname and a port. Being able to specify http connections with SRV resolution support is useful in contexts where external services are defined as SRV records.

Use case / motivation

One particular example applies to service discovery with dynamic port mapping. In this context, service discovery maps one or more hosts to a SRV record. When resolving that SRV record a client gets one or more hosts and ports to reach the desired service. In my particular use case I am using AWS Service Discovery for ECS with dynamic port mapping. I would like to reach my ECS service transparently from Airflow without going thru a load balancer.

I would like to be able to specify an http connection with a SRV record and let Airflow resolve the actual hostname and port using SRV lookup.

Related Issues

https://github.com/apache/airflow/pull/4481

I'll be happy to provide the implementation for this

Contributor guide

Research direction
Investigate how Airflow currently handles HTTP connections and URI parsing, then extend to support SRV DNS records by adding a new scheme or hook that resolves SRV records to host:port before establishing connections.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonAirflow
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65