[Spanner] Retries of UNAVAILABLE errors when resuming a result stream not working properly
Maintainers usually reply within 1 day
@Hectorhammett is already working on this.
Since Sep 25, 2026.
Assessment
This issue has not been assessed yet.
Description
Result::rows() can retry UNAVAILABLE errors forever, when the server keeps returning UNAVAILABLE after the first stream is interrupted. The retry limit from RetrySettings (default 3) is never applied.
This issue is a bit hard to explain but here goes...
The bit of code that needs fixing is here.
https://github.com/googleapis/google-cloud-php/blob/c891a48bd3bd3f132dd7bc3bc35ff73844743abb/Spanner/src/Result.php#L182
This block is called when a ServiceException is thrown with status UNAVAILABLE while the Generator returned from $call is iterating. I think the expected behavior here is to re-execute the $call, reconnect and resume the iterating process. At first glance it seemed like it's doing exactly that.
But when I actually run it, I noticed that when $call is actually invoked in ExponentialBackoff::execute($call, ...), it's just returning the Generator that is not started (see below).
Since the code inside the $call is not executed, the unstarted generator gets incorrectly assigned and the loop is continued. When $this->generator->current() is called again (Result.php:L138), and if another ServiceException (with status UNAVAILABLE) is thrown again, the whole process is repeated.
I think the expected behavior here is to have the generator start when $backoff->execute(...) is run so that backoff can actually be applied.
Result::createGenerator(...) run with a similar logic, but there the generator is started inside the callback so it runs correctly like so...
So I think this can be fixed by applying by changing...
$this->generator = $backoff->execute($call, [$this->resumeToken, $this->transaction()]);
to
$this->generator = $backoff->execute(function () use ($call) {
$generator = $call();
$generator->valid();
return $generator;
});
Environment details
- OS: Alpine Linux
- PHP version: 8.5.10
- Package name and version: google/cloud-spanner v2.11.0
Steps to reproduce
- Run a large query using
Database::execute(...) - Call
rows() - Start iterating.
- Network disconnects and throw UNAVAILABLE midway.
- Dominant language
- PHP
- Stars
- 1.2k
- Forks
- 464
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 81
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from googleapis/google-cloud-php
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
googleapis/google-cloud-php#9730 ·
Maintainers usually reply within 1 day
-
type: feature request
Difficulty 1/5 Under an hour Newbie friendliness 75/100
googleapis/google-cloud-php#9716 · 11 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
googleapis/google-cloud-php#9739 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
googleapis/google-cloud-php#9725 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
googleapis/google-cloud-php#9675 · 1 comment ·
Maintainers usually reply within 1 day
All issues in googleapis/google-cloud-php
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
domain/crm-after-sales Platform(Default) priority/high
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
kind/bug status/to verify
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
PHP-CS-Fixer/PHP-CS-Fixer#9867 ·
Maintainers usually reply within 1 day
-
sync-en
Difficulty 1/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 2 days
-
sync-en
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 2 days