Retry mechanism behaves incorrectly when HTTP 429 is returned by Datadog

まず、issue に記載されているダッシュボード取得シーケンスと retry 関連のテストを使って、クライアント内で enable_retry オプションが HTTP 429 応答をどのように処理するかを追跡します。レート制限のケースを再現し、最初の 429 で終了するのではなく、再試行する前にクライアントが x-ratelimit-reset まで待機することを確認します。

Describe the bug
Summary:
Script which fetches all dashboards in the loop exits with an error, when rate limit runs out, even though enable_retry option is turned on. According to debug logs, script exits on first HTTP 429 returned, with no retry attempted.

Details:
We have a script, which runs every night to fetch all dashboards from Datadog. It does it by fetching list of dashboards, and then goes one after another to fetch details of each one. After number of our dashboard grew, we have run into HTTP 429 errors due to Datadog's rate limit.

We decided to use retry option, which is built into the library since 2.16.0, but it seems it's not ready to how Datadog responds in case of hitting rate limit.

When I started the script in a loop with debug option enabled I see that Datadog returns HTTP 200 up until the moment when rate limit is reached, then next request gets HTTP 429 (API Keys removed from logs below):

# normal request before rate limit runs out
send: b'GET /api/v1/dashboard/<id-of-dashboard-59> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 1
header: x-ratelimit-reset: 29
header: x-ratelimit-name: dashboards_get_custom_api

# normal request, last one within the limits
send: b'GET /api/v1/dashboard/<id-of-dashboard-60> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# next request, this one gets HTTP 429
send: b'GET /api/v1/dashboard/<id-of-dashboard-61> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 429 Too Many Requests'
...
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# and at this point script fails with
Error: (429)
Reason: Too Many Requests
HTTP response headers: {'x-ratelimit-limit': '60', 'x-ratelimit-period': '60', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '28', 'x-ratelimit-name': 'dashboards_get_custom_api', 'content-type': 'application/json', 'Content-Length': '183', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'date': 'Mon, 25 Mar 2024 09:05:32 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
HTTP response body: {'status': 'error', 'code': 429, 'errors': ['Too many requests'], 'statuspage': 'http://status.us5.datadoghq.com', 'twitter': 'http://twitter.com/datadogops', 'email': 'support@datadoghq.com'}

To Reproduce
See description above

Expected behavior
I expect library to sleep for x-ratelimit-reset time, just like it's described in tests, which introduced this functionality. Right now I need to add a sleep between requests to API as a workaround

Screenshots
N/A - logs attached

Environment and Versions (please complete the following information):
client library version 2.23.0

Additional context
Add any other context about the problem here.