BrokenPipeError when using Attached Service Account on GCE Instance

Inizia con google_auth_httplib2.py e google/auth/compute_engine/_metadata.py nel percorso di rinnovo del token mostrato dal traceback. Riproduci il problema su un’istanza GCE dopo la scadenza delle credenziali mentre effettui una chiamata alla Drive API, quindi verifica che il rinnovo e la successiva richiesta API vengano completati senza BrokenPipeError.

Environment details

• OS type and version: Google Compute Engine (GCE) with Container-Optimized OS
• Python version: Python 3.7.16
• pip version: pip 21.3.1
• google-api-python-client== 1.12.8
• google-auth==1.35.0
• google-auth-httplib2==0.2.0

Steps to reproduce

1. Deploy a Python application on a GCE instance with an attached service account.
2. Use google.auth.default() for authentication.
3. Let the service run for a few hours to allow the token to expire.
4. Perform a Google Drive API call during token refresh (e.g., listing files or creating a folder).

Code example

gdrive_credentials: Credentials = (
    Credentials.from_service_account_info(
        gcp_credentials_dict,
        scopes=SCOPES,
    )
    if gcp_credentials_dict
    else google.auth.default(scopes=SCOPES)[0]
)

def create_services(credentials, scopes):
    drive_service = build(
        "drive", "v3", credentials=credentials
    )
    docs_service = build(
        "docs", "v1", credentials=credentials
    )

    return credentials, drive_service, docs_service

def get_or_create_folder(self, drive_service, folder_name):
    """
    Checks if a folder exists in the root directory and creates it if not.

    :param drive_service: Authorized Drive API client.
    :param folder_name: The name of the folder to check or create.
    :return: The folder ID.
    """
    # Search for the folder in the root directory
    query = f"name = '{folder_name}' and mimeType = 'application/vnd.google-apps.folder' and 'root' in parents and trashed = false"
    response = (
        drive_service.files().list(q=query, fields="files(id, name)").execute()
    )
    files = response.get("files", [])

    if files:
        # Folder exists, return its ID
        return files[0]["id"]
    else:
        # Folder doesn't exist, create it
        folder_metadata = {
            "name": folder_name,
            "mimeType": "application/vnd.google-apps.folder",
            "parents": ["root"],  # Place in the root directory
        }
        folder = (
            drive_service.files()
            .create(body=folder_metadata, fields="id")
            .execute()
        )
        return folder["id"]

Stacktrace

BrokenPipeError: [Errno 32] Broken pipe
  File "/app/endpoints/webhooks.py", line 292, in get_services_status
    current_app.config.get("BUILD_ENVIRONMENT"),
  File "/app/middleware/gdocsprinter.py", line 43, in print_file_with_template
    template_id, gdocs_drive_service, replace_template, build_environment
  File "/app/middleware/gdocsprinter.py", line 111, in create_template_copy
    drive_service, f"{build_environment}-prints"
  File "/app/middleware/gdocsprinter.py", line 86, in get_or_create_folder
    drive_service.files().list(q=query, fields="files(id, name)").execute()
  File "googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 909, in execute
    headers=self.headers,
  File "googleapiclient/http.py", line 204, in _retry_request
    raise exception
  File "googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "google_auth_httplib2.py", line 190, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 111, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 88, in _retrieve_info
    request, service_account=self._service_account_email
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 150, in get
    response = request(url=url, method="GET", headers=_METADATA_HEADERS)
  File "google_auth_httplib2.py", line 117, in __call__
    url, method=method, body=body, headers=headers, **kwargs)
  File "__init__.py", line 1709, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "__init__.py", line 1347, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "http/client.py", line 1036, in _send_output
    self.send(msg)
  File "http/client.py", line 997, in send
    self.sock.sendall(data)

Brief summary

After switching the authentication method for our application from service account keys to an attached service account on a Google Compute Engine (GCE) instance, we are encountering a BrokenPipeError during API calls to Google Drive.

The issue only occurs during token refresh after the application has been running for a couple of hours. Following the first failure, the service works again without intervention.

Thanks