BrokenPipeError when using Attached Service Account on GCE Instance

Open
  • Is something wrong?
#2,553 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
google-cloud, python
Domain
api, authentication, cloud

Research direction

Start with google_auth_httplib2.py and google/auth/compute_engine/_metadata.py in the token-refresh path shown by the traceback. Reproduce on a GCE instance after credential expiry while making a Drive API call, then verify the refresh and subsequent API request complete without BrokenPipeError.

Written by the indexing model from the issue text.

Description

priority: p3 type: bug
Environment details
  • OS type and version: Google Compute Engine (GCE) with Container-Optimized OS
  • Python version: Python 3.7.16
  • pip version: pip 21.3.1
  • google-api-python-client== 1.12.8
  • google-auth==1.35.0
  • google-auth-httplib2==0.2.0
Steps to reproduce
  1. Deploy a Python application on a GCE instance with an attached service account.
  2. Use google.auth.default() for authentication.
  3. Let the service run for a few hours to allow the token to expire.
  4. Perform a Google Drive API call during token refresh (e.g., listing files or creating a folder).
Code example
gdrive_credentials: Credentials = (
    Credentials.from_service_account_info(
        gcp_credentials_dict,
        scopes=SCOPES,
    )
    if gcp_credentials_dict
    else google.auth.default(scopes=SCOPES)[0]
)

def create_services(credentials, scopes):
    drive_service = build(
        "drive", "v3", credentials=credentials
    )
    docs_service = build(
        "docs", "v1", credentials=credentials
    )

    return credentials, drive_service, docs_service

def get_or_create_folder(self, drive_service, folder_name):
    """
    Checks if a folder exists in the root directory and creates it if not.

    :param drive_service: Authorized Drive API client.
    :param folder_name: The name of the folder to check or create.
    :return: The folder ID.
    """
    # Search for the folder in the root directory
    query = f"name = '{folder_name}' and mimeType = 'application/vnd.google-apps.folder' and 'root' in parents and trashed = false"
    response = (
        drive_service.files().list(q=query, fields="files(id, name)").execute()
    )
    files = response.get("files", [])

    if files:
        # Folder exists, return its ID
        return files[0]["id"]
    else:
        # Folder doesn't exist, create it
        folder_metadata = {
            "name": folder_name,
            "mimeType": "application/vnd.google-apps.folder",
            "parents": ["root"],  # Place in the root directory
        }
        folder = (
            drive_service.files()
            .create(body=folder_metadata, fields="id")
            .execute()
        )
        return folder["id"]
Stacktrace
BrokenPipeError: [Errno 32] Broken pipe
  File "/app/endpoints/webhooks.py", line 292, in get_services_status
    current_app.config.get("BUILD_ENVIRONMENT"),
  File "/app/middleware/gdocsprinter.py", line 43, in print_file_with_template
    template_id, gdocs_drive_service, replace_template, build_environment
  File "/app/middleware/gdocsprinter.py", line 111, in create_template_copy
    drive_service, f"{build_environment}-prints"
  File "/app/middleware/gdocsprinter.py", line 86, in get_or_create_folder
    drive_service.files().list(q=query, fields="files(id, name)").execute()
  File "googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 909, in execute
    headers=self.headers,
  File "googleapiclient/http.py", line 204, in _retry_request
    raise exception
  File "googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "google_auth_httplib2.py", line 190, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 111, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 88, in _retrieve_info
    request, service_account=self._service_account_email
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 150, in get
    response = request(url=url, method="GET", headers=_METADATA_HEADERS)
  File "google_auth_httplib2.py", line 117, in __call__
    url, method=method, body=body, headers=headers, **kwargs)
  File "__init__.py", line 1709, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "__init__.py", line 1347, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "http/client.py", line 1036, in _send_output
    self.send(msg)
  File "http/client.py", line 997, in send
    self.sock.sendall(data)
Brief summary

After switching the authentication method for our application from service account keys to an attached service account on a Google Compute Engine (GCE) instance, we are encountering a BrokenPipeError during API calls to Google Drive.

The issue only occurs during token refresh after the application has been running for a couple of hours. Following the first failure, the service works again without intervention.

Thanks

Dominant language
Python
Stars
9k
Forks
2.6k
Avg merge
28m
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from googleapis/google-api-python-client

All issues in googleapis/google-api-python-client

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.