Getting execution timeouts without clear reason why
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 30/100
- Tipo di issue
- Bug
- Chiarezza
- Da chiarire
- Stato di attività
- Attiva
- Stack tecnologico
- aws, php
- Ambito
- backend, cloud, networking
Direzione di ricerca
Start with the AWS SDK for PHP request flow described in the issue and vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php around line 169. Compare logged SDK requests with the corresponding CloudTrail events, focusing on the request that never arrives at AWS during rapid successive calls. Done means the timeout cause is identified or a reliably reproducible failure case is documented.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Describe the bug
I have a project with an automated testsuite. Many integration tests of the testsuite trigger the following request flow:
A. Test Client sends API Request to an application A hosted on a VPS X (sitting outside of AWS)
B. Application A sends a request to application B which is hosted on the same VPS X
C. Application B, which has the AWS SDK for PHP installed in it, gets data from AWS
D. AWS responds back to Application B on VPS X
E. Application B responds back to Application A also hosted on VPS X
F. Application A responds back to Test Client
Step C. above does two requests in sequence:
AssumeRole XYZ, then call the SDK'smemoizemethod for caching in case needed in same request- Simple request to get data from AWS, e.g. a key from
KMS, via assumed role
In the past weeks, the testsuite failed at what appeared to be a random integration test. Sometimes after around 3 mins of the test pipeline running, sometimes 7 mins, etc (For the concerned suite, tests are always executed in the same order.)
I investigated this for days and finally discovered that the reason for failure is always the exact same reason: If the application B executes the Steps 1. and 2. mentioned above multiple times in a row in quick succession, but still way below the rate / throttling / througput limit for the concerned services, the request from application B to AWS - after hanging for > 25s - fails with the following error:
Got error 'PHP message: PHP Fatal error: Maximum execution time of 30 seconds exceeded in /.../vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php](.../vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php) on line 169‘
I logged the requests causing the failure within guzzle and could in this way confirm that the request causing the timeouts are always one of the AWS SDK requests 1 or 2 mentioned above.
To try to solve this on my own:
- I updated the entire stack of the concerned application to PHP 8.5.10
- I updated the AWS SDK for PHP of the application to version 3.398.2
- I checked with the company managing the VPS X (I have no root access) if they could find ANYTHING - they could not, not even in the php logs only they have access to
- I checked that the concerned application server has no special server configs whatsoever
- I checked that the application error log which captures every failure happening on the application layer exclusively reports the above-mentioned error, nothing else
- I inspected all related CloudTrail logs on the AWS account to which the application B fires requests. And could find not a single failed event, not a single event with an error.
I then executed another full integration test run for which I logged all AWS SDK requests fired through guzzle from application B on my VPS X to AWS. Then I compared these logs with the corresponding CloudTrail log by mapping the requests sent from the SDK from application B to the corresponding events encountered in the AWS Cloud Trail. The mapping was successful for all events, with the only exception of a single event: One event that was fired from application B never arrived within AWS (at least there's no trace of it in CloudTrail). And that precisely was the request provoking the timeout error.
I reproduced the above-mentioned 3/3 times, to really make sure that the apparent reason for failure is indeed an AWS request never reaching AWS, if requests are fired in quick succession (automated integration tests, but again, well below any rate limits).
Of course I could simply increase the max execution time, as suggested by this post, but it does not make sense to me that a request sequence consisting of simply one AssumeRole call followed by another single request for nothing in the MB range like getting a decrypted key from KMS introduces such a high latency. So I figured the problem must be elsewhere, but did everything I could imagine to try to solve this and am pretty clueluess ATM.
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
As mentioned the integration test suite started to fail since a few weeks, without changes to the infra and / or server loads.
Current Behavior
See description above.
Reproduction Steps
Hard to reproduce, see above
Possible Solution
No response
Additional Information/Context
No response
SDK version used
3.398.2
Environment details (Version of PHP (php -v)? OS name and version, etc.)
PHP 8.5.10 (cli) (built: Aug 31 2026 09:10:18) (NTS) Copyright (c) The PHP Group Zend Engine v4.5.10, Copyright (c) Zend Technologies with Zend OPcache v8.5.10, Copyright (c), by Zend Technologies
- Lingua principale
- PHP
- Stelle
- 6.2k
- Fork
- 1.3k
- Merge medio
- 2g 4h
- PR unite (30g)
- 10
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di aws/aws-sdk-php
-
feature-request needs-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
aws/aws-sdk-php#3365 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 3/5 Mezza giornata Idoneità per principianti 45/100
aws/aws-sdk-php#3377 ·
I maintainer di solito rispondono entro 1 giorno
-
Event stream decoder drops the final event when reading it reaches EOFForse già presa @lazerg l’ha presa 3 giorni fa. Aperta
Difficoltà 3/5 Mezza giornata Idoneità per principianti 62/100
aws/aws-sdk-php#3374 ·
I maintainer di solito rispondono entro 1 giorno
-
Event stream decoder reports a checksum mismatch when the checksum is read in partsForse già presa @lazerg l’ha presa 6 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 25/100
aws/aws-sdk-php#3371 ·
I maintainer di solito rispondono entro 1 giorno
-
AssumeRoleCredentialProvider replaces non-RuntimeException failures with TypeErrorForse già presa @stobrien89 l’ha presa oggi. Apertabug pending-release
Difficoltà 2/5 1-3 ore Idoneità per principianti 25/100
aws/aws-sdk-php#3370 · 1 commento · 1 assegnatario ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di aws/aws-sdk-php
Issue simili
-
loginWithCustomTokenExchange() fails when an organization is configuredForse già presa @awss1i l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
auth0/auth0-PHP#883 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
-
type/bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 75/100
flarum/framework#5231 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
major-release
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 67/100
Intervention/image#1553 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
omegaup/omegaup#10353 · 1 commento ·
I maintainer di solito rispondono entro 5 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 61/100