storage 9.0.0: retries never stop and run about 1 ms apart (maxRetries ignored, delays read as milliseconds)
Maintainer antworten meist innerhalb von 1 Tag
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Anfängerfreundlichkeit
- 52/100
- Issue-Typ
- Bug
- Klarheit
- Klar beschrieben
- Aktivitätsstatus
- Aktiv
- Tech-Stack
- nodejs, typescript
- Bereich
- api
Rechercherichtung
Start in the StorageTransport makeRequest method, where the retryConfig object is built from maxRetries and retryOptions (src/storage-transport.ts, compiled to build/cjs/src/storage-transport.js). Check how the shouldRetry wrapper ignores the attempt count, and whether maxRetryDelay and totalTimeout are passed to gaxios in seconds when gaxios expects milliseconds. Run the reproduction script with a local server that returns 503. Done when the call rejects after about 4 requests and roughly 15 seconds, and no TimeoutNegativeWarning appears.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Package: @google-cloud/storage (handwritten/storage)
Environment
- @google-cloud/storage 9.0.0 (gaxios 8.3.3, google-auth-library 11.3.4)
- Node 26.5.1 (same under Bun 1.4)
- 8.0.0 behaves as documented with the same script
Steps to reproduce
A local server that always answers 503, so no bucket or credentials are needed:
import http from "node:http";
import { Storage } from "@google-cloud/storage";
import { JWT } from "google-auth-library";
let hits = 0;
const server = http.createServer((req, res) => {
hits++;
res.writeHead(503, { "content-type": "application/json" });
res.end('{"error":{"code":503,"message":"down"}}');
});
await new Promise((r) => server.listen(0, "127.0.0.1", r));
const authClient = new JWT({ email: "[email protected]", key: "fake" });
authClient.credentials = { access_token: "t", expiry_date: Date.now() + 3600e3 };
const storage = new Storage({ projectId: "p", authClient, apiEndpoint: `http://127.0.0.1:${server.address().port}` });
const t0 = Date.now();
const outcome = await Promise.race([
storage.bucket("b").file("o").getMetadata().then(() => "resolved", (e) => `rejected code=${e.code}`),
new Promise((r) => setTimeout(() => r("still pending"), 8000)),
]);
console.log({ outcome, ms: Date.now() - t0, requests: hits });
process.exit(0);
Actual (9.0.0)
TimeoutNegativeWarning: -2 is a negative number.
Timeout duration was set to 1.
{ outcome: 'still pending', ms: 8001, requests: 5039 }
The call never settles and sends about 600 requests per second. The same happens for a 429 on an idempotent upload (file.save(buffer, { resumable: false, preconditionOpts: { ifGenerationMatch: 0 } })), and a refused connection is still pending after 20 s.
Expected (8.0.0)
{ outcome: 'rejected code=503', ms: 15430, requests: 4 }
maxRetries (default 3) with exponential backoff, as retryOptions documents.
Cause
In StorageTransport.makeRequest (build/cjs/src/storage-transport.js):
retryConfig: {
retry: maxRetries,
noResponseRetries: maxRetries,
maxRetryDelay: this.retryOptions.maxRetryDelay,
retryDelayMultiplier: this.retryOptions.retryDelayMultiplier,
totalTimeout: this.retryOptions.totalTimeout,
shouldRetry: (err) => !isRetryDisabled && !!this.retryOptions.retryableErrorFn?.(err),
},
- No retry cap. In gaxios,
retryandnoResponseRetriesare only enforced inside the defaultshouldRetryRequest. Passing a customshouldRetryreplaces that function (config.shouldRetry || shouldRetryRequest), and the custom one never looks atcurrentRetryAttempt, so retries continue for as long as the error is retryable. - Seconds passed as milliseconds.
maxRetryDelay(64) andtotalTimeout(600) are documented in seconds, andresumable-upload.jsmultiplies them by 1000, but gaxios reads them as milliseconds. The backoff is therefore capped at 64 ms, and once 600 ms have passedtotalTimeout - elapsedis negative, which is where theTimeoutNegativeWarningand the 1 ms delay come from.
Impact
During a 5xx or 429 incident, or a network failure, every idempotent call hangs until the caller's own deadline while sending hundreds of requests per second to the API, instead of failing after about 15 seconds.
Suggested fix
Enforce the attempt cap in the shouldRetry wrapper (for example err.config.retryConfig.currentRetryAttempt < maxRetries), and convert maxRetryDelay and totalTimeout to milliseconds when building the gaxios retryConfig.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 3.2k
- Forks
- 732
- Ø Merge
- 2 T. 11 Std.
- Gemergte PRs (30 T.)
- 99
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus googleapis/google-cloud-node
-
Broken deprecation chainOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 66/100
googleapis/google-cloud-node#9854 ·
Maintainer antworten meist innerhalb von 1 Tag
-
pubsub: "Operation attempted on ended Span" when a modack is flushed after the message was ackedOffenapi: pubsub
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9853 ·
Maintainer antworten meist innerhalb von 1 Tag
-
storage: V4 getSignedUrl sorts canonical headers with localeCompare, so they disagree with X-Goog-SignedHeadersEvtl. vergeben @itsrayeedscodespace hat das vor 9 Tagen übernommen. Offenapi: storage
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9840 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
-
firestore: getPartitions() emits gax AutopaginateTrueWarning in every processEvtl. vergeben @TheRealDanii hat das vor 13 Tagen übernommen. Offenapi: firestore
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
googleapis/google-cloud-node#9736 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9723 ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in googleapis/google-cloud-node
Ähnliche Issues
-
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 83/100
gofish-graphics/gofish-graphics#1384 ·
Maintainer antworten meist innerhalb von 1 Tag
-
adapter bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 83/100
Olanetsoft/agent-lookout#353 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
vicharanashala/vibe#1542 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bug chore P2 size:XS
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 77/100
josephfung/curia#2666 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bug good first issue
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 84/100
devarshishimpi/codra#178 ·
Maintainer antworten meist innerhalb von 1 Tag