Sync GAPIC client construction performs blocking metadata-server DNS + HTTP via google.auth.default(), stalling asyncio callers

Ouverte
  • Quelque chose ne va pas ?
#18,412 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
authentication, backend, networking, performance

Piste de recherche

Start with pubsub_v1/services/publisher/transports/base.py, google/auth/_default.py, and google/auth/_metadata.py, then run the provided asyncio reproduction against the metadata-server path. Trace how PublisherClient.init reaches google.auth.default() and compare the documented alternatives. Done requires an agreed approach that addresses repeated construction and verifies that sync client creation no longer unexpectedly stalls asyncio callers.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

Constructing any sync GAPIC client with no explicit credentials performs blocking network I/O on the calling thread. When that thread is an asyncio event loop, the whole loop stalls. Nothing in the docstrings or docs says construction does I/O, and the recommended-looking usage ("instantiate this client with no arguments") is the one that triggers it.

Versions: google-cloud-pubsub 2.29.0, google-auth 2.39.0, google-api-core 2.24.2, Python 3.12.

Repro

import asyncio, time
from google.cloud import pubsub_v1

async def ticker():
    while True:
        t = time.perf_counter()
        await asyncio.sleep(0.01)
        lag = (time.perf_counter() - t - 0.01) * 1000
        if lag > 5:
            print(f"loop lag {lag:.0f}ms")

async def main():
    asyncio.create_task(ticker())
    await asyncio.sleep(0.2)
    pubsub_v1.PublisherClient()   # blocks the loop

asyncio.run(main())

On GCE/GKE with Workload Identity and no explicit credentials this reports tens to thousands of ms of loop lag, per construction. It reproduces on every sync GAPIC client, not just Pub/Sub.

Where the I/O happens

  1. pubsub_v1/publisher/client.py — Client.__init__ → super().__init__(**kwargs)
  2. pubsub_v1/services/publisher/transports/base.py:105-107 — elif credentials is None and not self._ignore_credentials: credentials, _ = google.auth.default(...)
  3. google/auth/_default.py:650 — runs the ADC checker chain; on GCE the first three checkers miss and it reaches _get_gce_credentials
  4. google/auth/_default.py:350 — _metadata.is_on_gce() → ping() → HTTP GET http://169.254.169.254 (_metadata.py:48-51), up to 3 attempts with exponential backoff
  5. google/auth/_default.py:353 — _metadata.get_project_id() → HTTP GET http://metadata.google.internal/computeMetadata/v1/project/project-id (_metadata.py:40-45), up to 5 attempts. This one resolves a hostname, so it includes a blocking getaddrinfo.
  6. google/auth/transport/_http_client.py:97 — each call builds a fresh http.client.HTTPConnection; no connection reuse, so every construction pays a new resolve and a new TCP connect.

There is no caching at any layer: google.auth.default() is re-run in full for every client constructed.

Worth noting for anyone trying to mitigate this with environment variables: setting GOOGLE_CLOUD_PROJECT does not avoid the project-id fetch. _get_gce_credentials calls get_project_id() unconditionally at _default.py:353, and explicit_project_id is only read at _default.py:635 and applied at _default.py:658 — after both HTTP calls have already happened. Only GCE_METADATA_HOST set to a literal IP removes the getaddrinfo from the path.

Impact (production)

A FastAPI/uvicorn service on GKE with Workload Identity constructed a PublisherClient() per publish — which looked safe, since nothing documents construction as doing I/O.

When cluster DNS began dropping answers, ndots:5 plus six search domains turned each metadata.google.internal lookup into 12+ queries, each dropped one costing a 5s glibc timeout. Every request stalled the single event loop; p99 latency rose roughly an order of magnitude for hours, and a meaningful fraction of requests timed out.

py-spy dump caught MainThread blocked in getaddrinfo inside PublisherClient.__init__ in a significant fraction of samples, and the process accumulated hundreds of threads from leaked clients.

That thread growth is a secondary effect of per-call construction: each fresh client starts its own batch-commit thread (pubsub_v1/publisher/client.py:530-543, _batch/thread.py:231-238) and leaks its gRPC channel.

Measurement: the cost is credential resolution, not gRPC

Constructing the client repeatedly with PUBSUB_EMULATOR_HOST set — which injects AnonymousCredentials at pubsub_v1/publisher/client.py:138 and skips ADC entirely — costs ~0.08 ms after warmup. The same construction with real ADC costs ~460 ms, and google.auth.default() alone accounts for essentially all of it.

gRPC channel creation is lazy-connect and effectively free. Credential resolution is the entire cost.

Why this is worth changing

Blocking network I/O in a constructor is invisible to callers and is a well-known hazard for async code. Two things make it worse here:

  • The docstring actively encourages the triggering pattern and never mentions I/O or reuse: "Generally, you can instantiate this client with no arguments, and you get sensible defaults."
  • It cannot be caught in testing. With PUBSUB_EMULATOR_HOST set, pubsub_v1/publisher/client.py injects AnonymousCredentials and google.auth.default() is never reached; locally, ADC resolves from a cached file. The metadata-server path only exists in production.

Ask

In order of preference:

  1. Defer credential resolution out of __init__ — resolve lazily on first RPC, where a caller can already expect I/O.
  2. Or memoize the ADC result process-wide, so repeated construction costs one resolution rather than N.
  3. Or, at minimum, document it: state in the PublisherClient / GAPIC client docstrings and in the auth docs that construction performs blocking network I/O (metadata-server HTTP plus a DNS lookup on GCE), and that clients should be constructed once and reused — explicitly off the event loop for asyncio callers.

Even (3) alone would have prevented this incident.

Related

  • googleapis/python-pubsub#1173 — metadata-server blip kills the publisher commit thread. Confirms synchronous metadata calls live in this path; no asyncio angle.
  • googleapis/python-pubsub#1279 — async client built outside a running loop hangs; the ADC check appears in the debug log but is not identified as the cause.
  • googleapis/google-auth-library-python#285 — "auth.default() takes 440ms". Pure perf observation, no blocking framing. Matches the ~460 ms measured above.
  • googleapis/google-auth-library-python#321 — "add asyncio support", the general sync-auth-stack-vs-async complaint; does not name client construction.

Filed here rather than against python-pubsub / google-auth-library-python / python-api-core, since those are archived and read-only. The behaviour spans packages/google-auth (where the resolution happens) and packages/google-cloud-pubsub (where it is triggered).

Langage dominant
Python
Étoiles
5.4k
Forks
1.8k
Merge moyen
2 j 7 h
PR mergées (30 j)
173

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de googleapis/google-cloud-python

Toutes les issues de googleapis/google-cloud-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.