Sync GAPIC client construction performs blocking metadata-server DNS + HTTP via google.auth.default(), stalling asyncio callers

Offen
  • Stimmt etwas nicht?
#18,412 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
authentication, backend, networking, performance

Rechercherichtung

Start with pubsub_v1/services/publisher/transports/base.py, google/auth/_default.py, and google/auth/_metadata.py, then run the provided asyncio reproduction against the metadata-server path. Trace how PublisherClient.init reaches google.auth.default() and compare the documented alternatives. Done requires an agreed approach that addresses repeated construction and verifies that sync client creation no longer unexpectedly stalls asyncio callers.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Summary

Constructing any sync GAPIC client with no explicit credentials performs blocking network I/O on the calling thread. When that thread is an asyncio event loop, the whole loop stalls. Nothing in the docstrings or docs says construction does I/O, and the recommended-looking usage ("instantiate this client with no arguments") is the one that triggers it.

Versions: google-cloud-pubsub 2.29.0, google-auth 2.39.0, google-api-core 2.24.2, Python 3.12.

Repro

import asyncio, time
from google.cloud import pubsub_v1

async def ticker():
    while True:
        t = time.perf_counter()
        await asyncio.sleep(0.01)
        lag = (time.perf_counter() - t - 0.01) * 1000
        if lag > 5:
            print(f"loop lag {lag:.0f}ms")

async def main():
    asyncio.create_task(ticker())
    await asyncio.sleep(0.2)
    pubsub_v1.PublisherClient()   # blocks the loop

asyncio.run(main())

On GCE/GKE with Workload Identity and no explicit credentials this reports tens to thousands of ms of loop lag, per construction. It reproduces on every sync GAPIC client, not just Pub/Sub.

Where the I/O happens

  1. pubsub_v1/publisher/client.py — Client.__init__ → super().__init__(**kwargs)
  2. pubsub_v1/services/publisher/transports/base.py:105-107 — elif credentials is None and not self._ignore_credentials: credentials, _ = google.auth.default(...)
  3. google/auth/_default.py:650 — runs the ADC checker chain; on GCE the first three checkers miss and it reaches _get_gce_credentials
  4. google/auth/_default.py:350 — _metadata.is_on_gce() → ping() → HTTP GET http://169.254.169.254 (_metadata.py:48-51), up to 3 attempts with exponential backoff
  5. google/auth/_default.py:353 — _metadata.get_project_id() → HTTP GET http://metadata.google.internal/computeMetadata/v1/project/project-id (_metadata.py:40-45), up to 5 attempts. This one resolves a hostname, so it includes a blocking getaddrinfo.
  6. google/auth/transport/_http_client.py:97 — each call builds a fresh http.client.HTTPConnection; no connection reuse, so every construction pays a new resolve and a new TCP connect.

There is no caching at any layer: google.auth.default() is re-run in full for every client constructed.

Worth noting for anyone trying to mitigate this with environment variables: setting GOOGLE_CLOUD_PROJECT does not avoid the project-id fetch. _get_gce_credentials calls get_project_id() unconditionally at _default.py:353, and explicit_project_id is only read at _default.py:635 and applied at _default.py:658 — after both HTTP calls have already happened. Only GCE_METADATA_HOST set to a literal IP removes the getaddrinfo from the path.

Impact (production)

A FastAPI/uvicorn service on GKE with Workload Identity constructed a PublisherClient() per publish — which looked safe, since nothing documents construction as doing I/O.

When cluster DNS began dropping answers, ndots:5 plus six search domains turned each metadata.google.internal lookup into 12+ queries, each dropped one costing a 5s glibc timeout. Every request stalled the single event loop; p99 latency rose roughly an order of magnitude for hours, and a meaningful fraction of requests timed out.

py-spy dump caught MainThread blocked in getaddrinfo inside PublisherClient.__init__ in a significant fraction of samples, and the process accumulated hundreds of threads from leaked clients.

That thread growth is a secondary effect of per-call construction: each fresh client starts its own batch-commit thread (pubsub_v1/publisher/client.py:530-543, _batch/thread.py:231-238) and leaks its gRPC channel.

Measurement: the cost is credential resolution, not gRPC

Constructing the client repeatedly with PUBSUB_EMULATOR_HOST set — which injects AnonymousCredentials at pubsub_v1/publisher/client.py:138 and skips ADC entirely — costs ~0.08 ms after warmup. The same construction with real ADC costs ~460 ms, and google.auth.default() alone accounts for essentially all of it.

gRPC channel creation is lazy-connect and effectively free. Credential resolution is the entire cost.

Why this is worth changing

Blocking network I/O in a constructor is invisible to callers and is a well-known hazard for async code. Two things make it worse here:

  • The docstring actively encourages the triggering pattern and never mentions I/O or reuse: "Generally, you can instantiate this client with no arguments, and you get sensible defaults."
  • It cannot be caught in testing. With PUBSUB_EMULATOR_HOST set, pubsub_v1/publisher/client.py injects AnonymousCredentials and google.auth.default() is never reached; locally, ADC resolves from a cached file. The metadata-server path only exists in production.

Ask

In order of preference:

  1. Defer credential resolution out of __init__ — resolve lazily on first RPC, where a caller can already expect I/O.
  2. Or memoize the ADC result process-wide, so repeated construction costs one resolution rather than N.
  3. Or, at minimum, document it: state in the PublisherClient / GAPIC client docstrings and in the auth docs that construction performs blocking network I/O (metadata-server HTTP plus a DNS lookup on GCE), and that clients should be constructed once and reused — explicitly off the event loop for asyncio callers.

Even (3) alone would have prevented this incident.

Related

  • googleapis/python-pubsub#1173 — metadata-server blip kills the publisher commit thread. Confirms synchronous metadata calls live in this path; no asyncio angle.
  • googleapis/python-pubsub#1279 — async client built outside a running loop hangs; the ADC check appears in the debug log but is not identified as the cause.
  • googleapis/google-auth-library-python#285 — "auth.default() takes 440ms". Pure perf observation, no blocking framing. Matches the ~460 ms measured above.
  • googleapis/google-auth-library-python#321 — "add asyncio support", the general sync-auth-stack-vs-async complaint; does not name client construction.

Filed here rather than against python-pubsub / google-auth-library-python / python-api-core, since those are archived and read-only. The behaviour spans packages/google-auth (where the resolution happens) and packages/google-cloud-pubsub (where it is triggered).

Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
173

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus googleapis/google-cloud-python

Alle Issues in googleapis/google-cloud-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.