[Auth]: Failed to read new token file after symlink rotation
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- php
- Domain
- authentication
Research direction
Start with src/CredentialSource/FileSource.php and its fetchSubjectToken() path, then inspect src/Credentials/ExternalAccountCredentials.php and the AWS SDK retry behavior linked in the issue. Reproduce the failure by rotating a symlink target within one PHP process, and consider how to verify that the current token is read successfully after rotation without exposing a transient authentication failure.
Written by the indexing model from the issue text.
Description
This is a client library issue. The product (STS token exchange) behaves correctly. The failure is in how Google\Auth\CredentialSource\FileSource reads the subject token file.
Environment details
- Programming language: PHP
- OS: Linux (Debian-based container) on Amazon EKS
- Language runtime version: PHP 8.4 (default
realpath_cache_ttl= 120,realpath_cache_size= 4096K) - Package version:
google/authv1.50.0. The code is unchanged in v1.53.0:
https://github.com/googleapis/google-auth-library-php/blob/v1.53.0/src/CredentialSource/FileSource.php
Summary
We use Workload Identity Federation from EKS. The Kubernetes service account token is mounted as a projected volume and referenced from the credential config with credential_source.file.
Kubelet rotates a projected token with an atomic symlink swap: it writes the new token into a new ..YYYY_MM_DD_... directory, repoints the ..data symlink, and deletes the old directory. PHP caches the resolved path of credential_source.file in its realpath cache for up to realpath_cache_ttl seconds (120 by default). After the swap, file_get_contents() on the unchanged path resolves to the deleted directory and fails.
FileSource::fetchSubjectToken() does a single file_get_contents($this->file) with no clearstatcache() and no retry. Every token refresh in that window fails with:
file_get_contents(/var/run/secrets/tokens/eks-token): Failed to open stream: No such file or directory
In comparison, the AWS SDK for PHP handles the same event explicitly for IRSA. On a failed read it clears the stat cache for the symlink target, the target directory, and the path, then re-reads with bounded retries:
https://github.com/aws/aws-sdk-php/blob/3.374.2/src/Credentials/AssumeRoleWithWebIdentityCredentialProvider.php#L103-L114
This is why the AWS token on the same pods does not have the problem.
Steps to reproduce
- Run a long-lived PHP 8.4 process (for example PHP-FPM) on Kubernetes with a projected service account token volume and a WIF credential config that points
credential_source.fileat it. Set a shortexpirationSeconds(for example 600) so kubelet rotates the token often. - Configure
ExternalAccountCredentials(orApplicationDefaultCredentialswithGOOGLE_APPLICATION_CREDENTIALSset to the config) and make requests continuously so the credential is refreshed. - Wait for kubelet to rotate the token. Kubelet rotates when the token passes 80% of its lifetime or 24 hours, whichever comes first.
- Observe
fetchSubjectToken()fail withNo such file or directoryfor up to 120 seconds after the rotation, until the realpath cache entry expires.
A unit-level reproduction that does not need Kubernetes: read a file through a symlink once with file_get_contents(), replace the symlink target with a new directory, delete the old directory, and read the same path again inside the same process without calling clearstatcache().
Expected behaviour
FileSource reads the current token after a rotation. A single failed read of a path that exists should not surface to the caller as an authentication failure.
Proposed fix
Two changes would resolve this. Either one alone would help.
-
Make
FileSource::fetchSubjectToken()resilient to the swap. On a failed read:readlink()the path,clearstatcache(true, ...)on the resolved target, its parent directory, and the path itself, then re-read. Retry a bounded number of times with backoff when the read still fails or returns an empty string. This mirrors the AWS SDK implementation linked above. -
Allow callers to supply their own
ExternalAccountCredentialSourceInterface. TodayExternalAccountCredentials::__construct()calls the privatebuildCredentialSource()which always doesnew FileSource(...). There is no constructor option, setter, or factory hook. We had to use reflection to swap the privatesubjectTokenFetcherproperty, which is fragile:
https://github.com/googleapis/google-auth-library-php/blob/v1.53.0/src/Credentials/ExternalAccountCredentials.php
Workaround we use today
We implemented an ExternalAccountCredentialSourceInterface that clears the stat cache and retries, and injected it via reflection into ExternalAccountCredentials. We also raised the token expirationSeconds to 86400 to limit rotations to one per day per pod. This reduces exposure but does not remove the race.
We are happy to open a pull request for either or both of the proposed changes if maintainers agree with the approach.
- Dominant language
- PHP
- Stars
- 1.2k
- Forks
- 464
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 103
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from googleapis/google-cloud-php
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
googleapis/google-cloud-php#9730 ·
-
type: feature request
Difficulty 1/5 Under an hour Newbie friendliness 75/100
googleapis/google-cloud-php#9716 · 11 reactions ·
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
googleapis/google-cloud-php#9725 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
googleapis/google-cloud-php#9675 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
googleapis/google-cloud-php#9503 ·
All issues in googleapis/google-cloud-php
Similar issues
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
UX
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
ProfessionalWiki/NeoWiki#1525 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
OpenConext/OpenConext-engineblock#2122 ·
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Automattic/safe-publish#594 ·