storage 9.0.0: retries never stop and run about 1 ms apart (maxRetries ignored, delays read as milliseconds)

オープン
#9,871 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
52/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
nodejs, typescript
領域
api

調査の方向性

Start in the StorageTransport makeRequest method, where the retryConfig object is built from maxRetries and retryOptions (src/storage-transport.ts, compiled to build/cjs/src/storage-transport.js). Check how the shouldRetry wrapper ignores the attempt count, and whether maxRetryDelay and totalTimeout are passed to gaxios in seconds when gaxios expects milliseconds. Run the reproduction script with a local server that returns 503. Done when the call rejects after about 4 requests and roughly 15 seconds, and no TimeoutNegativeWarning appears.

索引モデルが issue の本文から書いたものです。

説明

Package: @google-cloud/storage (handwritten/storage)

Environment

  • @google-cloud/storage 9.0.0 (gaxios 8.3.3, google-auth-library 11.3.4)
  • Node 26.5.1 (same under Bun 1.4)
  • 8.0.0 behaves as documented with the same script

Steps to reproduce

A local server that always answers 503, so no bucket or credentials are needed:

import http from "node:http";
import { Storage } from "@google-cloud/storage";
import { JWT } from "google-auth-library";

let hits = 0;
const server = http.createServer((req, res) => {
  hits++;
  res.writeHead(503, { "content-type": "application/json" });
  res.end('{"error":{"code":503,"message":"down"}}');
});
await new Promise((r) => server.listen(0, "127.0.0.1", r));

const authClient = new JWT({ email: "[email protected]", key: "fake" });
authClient.credentials = { access_token: "t", expiry_date: Date.now() + 3600e3 };
const storage = new Storage({ projectId: "p", authClient, apiEndpoint: `http://127.0.0.1:${server.address().port}` });

const t0 = Date.now();
const outcome = await Promise.race([
  storage.bucket("b").file("o").getMetadata().then(() => "resolved", (e) => `rejected code=${e.code}`),
  new Promise((r) => setTimeout(() => r("still pending"), 8000)),
]);
console.log({ outcome, ms: Date.now() - t0, requests: hits });
process.exit(0);

Actual (9.0.0)

TimeoutNegativeWarning: -2 is a negative number.
Timeout duration was set to 1.
{ outcome: 'still pending', ms: 8001, requests: 5039 }

The call never settles and sends about 600 requests per second. The same happens for a 429 on an idempotent upload (file.save(buffer, { resumable: false, preconditionOpts: { ifGenerationMatch: 0 } })), and a refused connection is still pending after 20 s.

Expected (8.0.0)

{ outcome: 'rejected code=503', ms: 15430, requests: 4 }

maxRetries (default 3) with exponential backoff, as retryOptions documents.

Cause

In StorageTransport.makeRequest (build/cjs/src/storage-transport.js):

retryConfig: {
    retry: maxRetries,
    noResponseRetries: maxRetries,
    maxRetryDelay: this.retryOptions.maxRetryDelay,
    retryDelayMultiplier: this.retryOptions.retryDelayMultiplier,
    totalTimeout: this.retryOptions.totalTimeout,
    shouldRetry: (err) => !isRetryDisabled && !!this.retryOptions.retryableErrorFn?.(err),
},
  1. No retry cap. In gaxios, retry and noResponseRetries are only enforced inside the default shouldRetryRequest. Passing a custom shouldRetry replaces that function (config.shouldRetry || shouldRetryRequest), and the custom one never looks at currentRetryAttempt, so retries continue for as long as the error is retryable.
  2. Seconds passed as milliseconds. maxRetryDelay (64) and totalTimeout (600) are documented in seconds, and resumable-upload.js multiplies them by 1000, but gaxios reads them as milliseconds. The backoff is therefore capped at 64 ms, and once 600 ms have passed totalTimeout - elapsed is negative, which is where the TimeoutNegativeWarning and the 1 ms delay come from.

Impact

During a 5xx or 429 incident, or a network failure, every idempotent call hangs until the caller's own deadline while sending hundreds of requests per second to the API, instead of failing after about 15 seconds.

Suggested fix

Enforce the attempt cap in the shouldRetry wrapper (for example err.config.retryConfig.currentRetryAttempt < maxRetries), and convert maxRetryDelay and totalTimeout to milliseconds when building the gaxios retryConfig.

主要言語
TypeScript
スター
3.2k
フォーク
732
平均マージ
2日 11時間
マージ済み PR(30日)
99

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

googleapis/google-cloud-node のほかの issue

googleapis/google-cloud-node の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。