Sync GAPIC client construction performs blocking metadata-server DNS + HTTP via google.auth.default(), stalling asyncio callers

Aperta
  • Qualcosa non va?
#18,412 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
authentication, backend, networking, performance

Direzione di ricerca

Start with pubsub_v1/services/publisher/transports/base.py, google/auth/_default.py, and google/auth/_metadata.py, then run the provided asyncio reproduction against the metadata-server path. Trace how PublisherClient.init reaches google.auth.default() and compare the documented alternatives. Done requires an agreed approach that addresses repeated construction and verifies that sync client creation no longer unexpectedly stalls asyncio callers.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Constructing any sync GAPIC client with no explicit credentials performs blocking network I/O on the calling thread. When that thread is an asyncio event loop, the whole loop stalls. Nothing in the docstrings or docs says construction does I/O, and the recommended-looking usage ("instantiate this client with no arguments") is the one that triggers it.

Versions: google-cloud-pubsub 2.29.0, google-auth 2.39.0, google-api-core 2.24.2, Python 3.12.

Repro

import asyncio, time
from google.cloud import pubsub_v1

async def ticker():
    while True:
        t = time.perf_counter()
        await asyncio.sleep(0.01)
        lag = (time.perf_counter() - t - 0.01) * 1000
        if lag > 5:
            print(f"loop lag {lag:.0f}ms")

async def main():
    asyncio.create_task(ticker())
    await asyncio.sleep(0.2)
    pubsub_v1.PublisherClient()   # blocks the loop

asyncio.run(main())

On GCE/GKE with Workload Identity and no explicit credentials this reports tens to thousands of ms of loop lag, per construction. It reproduces on every sync GAPIC client, not just Pub/Sub.

Where the I/O happens

  1. pubsub_v1/publisher/client.py — Client.__init__ → super().__init__(**kwargs)
  2. pubsub_v1/services/publisher/transports/base.py:105-107 — elif credentials is None and not self._ignore_credentials: credentials, _ = google.auth.default(...)
  3. google/auth/_default.py:650 — runs the ADC checker chain; on GCE the first three checkers miss and it reaches _get_gce_credentials
  4. google/auth/_default.py:350 — _metadata.is_on_gce() → ping() → HTTP GET http://169.254.169.254 (_metadata.py:48-51), up to 3 attempts with exponential backoff
  5. google/auth/_default.py:353 — _metadata.get_project_id() → HTTP GET http://metadata.google.internal/computeMetadata/v1/project/project-id (_metadata.py:40-45), up to 5 attempts. This one resolves a hostname, so it includes a blocking getaddrinfo.
  6. google/auth/transport/_http_client.py:97 — each call builds a fresh http.client.HTTPConnection; no connection reuse, so every construction pays a new resolve and a new TCP connect.

There is no caching at any layer: google.auth.default() is re-run in full for every client constructed.

Worth noting for anyone trying to mitigate this with environment variables: setting GOOGLE_CLOUD_PROJECT does not avoid the project-id fetch. _get_gce_credentials calls get_project_id() unconditionally at _default.py:353, and explicit_project_id is only read at _default.py:635 and applied at _default.py:658 — after both HTTP calls have already happened. Only GCE_METADATA_HOST set to a literal IP removes the getaddrinfo from the path.

Impact (production)

A FastAPI/uvicorn service on GKE with Workload Identity constructed a PublisherClient() per publish — which looked safe, since nothing documents construction as doing I/O.

When cluster DNS began dropping answers, ndots:5 plus six search domains turned each metadata.google.internal lookup into 12+ queries, each dropped one costing a 5s glibc timeout. Every request stalled the single event loop; p99 latency rose roughly an order of magnitude for hours, and a meaningful fraction of requests timed out.

py-spy dump caught MainThread blocked in getaddrinfo inside PublisherClient.__init__ in a significant fraction of samples, and the process accumulated hundreds of threads from leaked clients.

That thread growth is a secondary effect of per-call construction: each fresh client starts its own batch-commit thread (pubsub_v1/publisher/client.py:530-543, _batch/thread.py:231-238) and leaks its gRPC channel.

Measurement: the cost is credential resolution, not gRPC

Constructing the client repeatedly with PUBSUB_EMULATOR_HOST set — which injects AnonymousCredentials at pubsub_v1/publisher/client.py:138 and skips ADC entirely — costs ~0.08 ms after warmup. The same construction with real ADC costs ~460 ms, and google.auth.default() alone accounts for essentially all of it.

gRPC channel creation is lazy-connect and effectively free. Credential resolution is the entire cost.

Why this is worth changing

Blocking network I/O in a constructor is invisible to callers and is a well-known hazard for async code. Two things make it worse here:

  • The docstring actively encourages the triggering pattern and never mentions I/O or reuse: "Generally, you can instantiate this client with no arguments, and you get sensible defaults."
  • It cannot be caught in testing. With PUBSUB_EMULATOR_HOST set, pubsub_v1/publisher/client.py injects AnonymousCredentials and google.auth.default() is never reached; locally, ADC resolves from a cached file. The metadata-server path only exists in production.

Ask

In order of preference:

  1. Defer credential resolution out of __init__ — resolve lazily on first RPC, where a caller can already expect I/O.
  2. Or memoize the ADC result process-wide, so repeated construction costs one resolution rather than N.
  3. Or, at minimum, document it: state in the PublisherClient / GAPIC client docstrings and in the auth docs that construction performs blocking network I/O (metadata-server HTTP plus a DNS lookup on GCE), and that clients should be constructed once and reused — explicitly off the event loop for asyncio callers.

Even (3) alone would have prevented this incident.

Related

  • googleapis/python-pubsub#1173 — metadata-server blip kills the publisher commit thread. Confirms synchronous metadata calls live in this path; no asyncio angle.
  • googleapis/python-pubsub#1279 — async client built outside a running loop hangs; the ADC check appears in the debug log but is not identified as the cause.
  • googleapis/google-auth-library-python#285 — "auth.default() takes 440ms". Pure perf observation, no blocking framing. Matches the ~460 ms measured above.
  • googleapis/google-auth-library-python#321 — "add asyncio support", the general sync-auth-stack-vs-async complaint; does not name client construction.

Filed here rather than against python-pubsub / google-auth-library-python / python-api-core, since those are archived and read-only. The behaviour spans packages/google-auth (where the resolution happens) and packages/google-cloud-pubsub (where it is triggered).

Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
2g 7h
PR unite (30g)
173

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di googleapis/google-cloud-python

Tutte le issue di googleapis/google-cloud-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.