Dual-stack Addrinfo.udp lookup + Kubernetes ndots causes DNS query amplification against IPv4-only hosts

Start in lib/datadog/statsd/udp_connection.rb at Connection#connect and inspect how Addrinfo.udp(host, port).afamily performs resolution. Reproduce the lookup with an IPv4-only Kubernetes Service and configured search domains, using CoreDNS query logs to compare the dual-stack and single-family cases. Done should include an agreed mitigation and verification that the amplified fallback queries no longer occur or are clearly documented.

Summary

Connection#connect (lib/datadog/statsd/udp_connection.rb) resolves the agent host via Addrinfo.udp(host, port).afamily, which performs a dual-stack (AF_UNSPEC) getaddrinfo lookup — i.e. it queries both A and AAAA regardless of which family is actually needed. This was introduced in #280 (merged in v5.6.1) to support IPv6 agents, and the PR description already flagged the tradeoff:

This will additionally work for DNS hostnames, but it will add a bit of time penalty due to the double-DNS lookup.

In a Kubernetes environment with a search domain list and ndots configured (the default Pod DNS setup — see Kubernetes DNS docs), this "bit of time penalty" becomes a genuine query-volume amplifier when the agent host is IPv4-only, which is the common case (e.g. the Datadog Cluster Agent's Kubernetes Service typically has no AAAA record at all).

Root cause

ndots is documented to try the absolute (fully-qualified) form of a name first, falling back to search-domain-suffixed candidates only if that lookup fails. We found that a NODATA response (NOERROR, zero records) for one of the two dual-stack query types gets treated the same as an outright failure, re-triggering the search-domain fallback that's meant for genuine resolution failure (NXDOMAIN).

Confirmed live against three cases, each isolated with a distinct/never-queried hostname where relevant, capturing the actual DNS queries issued (CoreDNS query log):

So ndots isn't malfunctioning — the interaction is between its NXDOMAIN-triggers-fallback semantics and a dual-stack lookup where one address family legitimately has no records. Since the agent host is IPv4-only, every single resolution attempt against it hits this path, generating 4x the DNS query volume of a single-family lookup (and unlike a single-family lookup, half of every pair (the AAAA half) can structurally never succeed).

Impact

In our cluster, this specific hostname alone accounted for ~77% of all NXDOMAIN responses observed on a single CoreDNS replica in a 90-second capture window, and cluster-wide NXDOMAIN volume sustained ~160K–188K/minute. Not all of that is attributable to this gem specifically (other resolvers/services contribute too), but the dual-stack behavior here measurably doubles the query cost versus a single-family lookup for every dogstatsd client instantiation against a hostname (vs. IP literal).

Suggested mitigations (any of these would help; not asking for all)

1. Document the interaction with Kubernetes ndots/search domains explicitly, so users can proactively lower ndots or use a fully-qualified (trailing-dot) hostname to avoid it.
2. Offer a config option to force a single address family (e.g. family: :INET) when the caller knows the agent is IPv4-only, skipping the dual-stack lookup entirely.
3. Cache/reuse the resolved Addrinfo across reconnects more aggressively (if not already done) to reduce how often the amplified lookup recurs.

Environment

• dogstatsd-ruby 5.6.1
• Kubernetes (EKS), Pod DNS with ndots:2 explicitly configured (also reproduced conceptually at the cluster default ndots:5)
• Ruby 3.4.9

Happy to provide more detail (exact resolv.conf, raw CoreDNS log excerpts) if useful. Also noticed #287 (open, same version, Sidekiq integration silently losing metrics on EKS after upgrading to 5.6.1) — I can't confirm it shares this root cause, but the version/platform overlap seemed worth mentioning there too.