Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Auth]: Failed to read new token file after symlink rotation

Open
#9,674 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
php

Research direction

Start with src/CredentialSource/FileSource.php and its fetchSubjectToken() path, then inspect src/Credentials/ExternalAccountCredentials.php and the AWS SDK retry behavior linked in the issue. Reproduce the failure by rotating a symlink target within one PHP process, and consider how to verify that the current token is read successfully after rotation without exposing a transient authentication failure.

Written by the indexing model from the issue text.

Description

This is a client library issue. The product (STS token exchange) behaves correctly. The failure is in how Google\Auth\CredentialSource\FileSource reads the subject token file.

Environment details
Summary

We use Workload Identity Federation from EKS. The Kubernetes service account token is mounted as a projected volume and referenced from the credential config with credential_source.file.

Kubelet rotates a projected token with an atomic symlink swap: it writes the new token into a new ..YYYY_MM_DD_... directory, repoints the ..data symlink, and deletes the old directory. PHP caches the resolved path of credential_source.file in its realpath cache for up to realpath_cache_ttl seconds (120 by default). After the swap, file_get_contents() on the unchanged path resolves to the deleted directory and fails.

FileSource::fetchSubjectToken() does a single file_get_contents($this->file) with no clearstatcache() and no retry. Every token refresh in that window fails with:

file_get_contents(/var/run/secrets/tokens/eks-token): Failed to open stream: No such file or directory

In comparison, the AWS SDK for PHP handles the same event explicitly for IRSA. On a failed read it clears the stat cache for the symlink target, the target directory, and the path, then re-reads with bounded retries:
https://github.com/aws/aws-sdk-php/blob/3.374.2/src/Credentials/AssumeRoleWithWebIdentityCredentialProvider.php#L103-L114

This is why the AWS token on the same pods does not have the problem.

Steps to reproduce
  1. Run a long-lived PHP 8.4 process (for example PHP-FPM) on Kubernetes with a projected service account token volume and a WIF credential config that points credential_source.file at it. Set a short expirationSeconds (for example 600) so kubelet rotates the token often.
  2. Configure ExternalAccountCredentials (or ApplicationDefaultCredentials with GOOGLE_APPLICATION_CREDENTIALS set to the config) and make requests continuously so the credential is refreshed.
  3. Wait for kubelet to rotate the token. Kubelet rotates when the token passes 80% of its lifetime or 24 hours, whichever comes first.
  4. Observe fetchSubjectToken() fail with No such file or directory for up to 120 seconds after the rotation, until the realpath cache entry expires.

A unit-level reproduction that does not need Kubernetes: read a file through a symlink once with file_get_contents(), replace the symlink target with a new directory, delete the old directory, and read the same path again inside the same process without calling clearstatcache().

Expected behaviour

FileSource reads the current token after a rotation. A single failed read of a path that exists should not surface to the caller as an authentication failure.

Proposed fix

Two changes would resolve this. Either one alone would help.

  1. Make FileSource::fetchSubjectToken() resilient to the swap. On a failed read: readlink() the path, clearstatcache(true, ...) on the resolved target, its parent directory, and the path itself, then re-read. Retry a bounded number of times with backoff when the read still fails or returns an empty string. This mirrors the AWS SDK implementation linked above.

  2. Allow callers to supply their own ExternalAccountCredentialSourceInterface. Today ExternalAccountCredentials::__construct() calls the private buildCredentialSource() which always does new FileSource(...). There is no constructor option, setter, or factory hook. We had to use reflection to swap the private subjectTokenFetcher property, which is fragile:
    https://github.com/googleapis/google-auth-library-php/blob/v1.53.0/src/Credentials/ExternalAccountCredentials.php

Workaround we use today

We implemented an ExternalAccountCredentialSourceInterface that clears the stat cache and retries, and injected it via reflection into ExternalAccountCredentials. We also raised the token expirationSeconds to 86400 to limit rotations to one per day per pod. This reduces exposure but does not remove the race.

We are happy to open a pull request for either or both of the proposed changes if maintainers agree with the approach.

Dominant language
PHP
Stars
1.2k
Forks
464
Avg merge
2d 2h
Merged PRs (30d)
103

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from googleapis/google-cloud-php

All issues in googleapis/google-cloud-php

Similar issues

More PHP issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.