Retry mechanism behaves incorrectly when HTTP 429 is returned by Datadog

Inizia tracciando il modo in cui l’opzione enable_retry gestisce le risposte HTTP 429 nel client, usando la sequenza di recupero della dashboard e i test relativi ai tentativi descritti nell’issue. Riproduci il caso di limitazione della frequenza e verifica che il client attenda x-ratelimit-reset prima di ritentare, invece di terminare al primo 429.

Describe the bug
Summary:
Script which fetches all dashboards in the loop exits with an error, when rate limit runs out, even though enable_retry option is turned on. According to debug logs, script exits on first HTTP 429 returned, with no retry attempted.

Details:
We have a script, which runs every night to fetch all dashboards from Datadog. It does it by fetching list of dashboards, and then goes one after another to fetch details of each one. After number of our dashboard grew, we have run into HTTP 429 errors due to Datadog's rate limit.

We decided to use retry option, which is built into the library since 2.16.0, but it seems it's not ready to how Datadog responds in case of hitting rate limit.

When I started the script in a loop with debug option enabled I see that Datadog returns HTTP 200 up until the moment when rate limit is reached, then next request gets HTTP 429 (API Keys removed from logs below):

# normal request before rate limit runs out
send: b'GET /api/v1/dashboard/<id-of-dashboard-59> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 1
header: x-ratelimit-reset: 29
header: x-ratelimit-name: dashboards_get_custom_api

# normal request, last one within the limits
send: b'GET /api/v1/dashboard/<id-of-dashboard-60> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# next request, this one gets HTTP 429
send: b'GET /api/v1/dashboard/<id-of-dashboard-61> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 429 Too Many Requests'
...
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# and at this point script fails with
Error: (429)
Reason: Too Many Requests
HTTP response headers: {'x-ratelimit-limit': '60', 'x-ratelimit-period': '60', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '28', 'x-ratelimit-name': 'dashboards_get_custom_api', 'content-type': 'application/json', 'Content-Length': '183', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'date': 'Mon, 25 Mar 2024 09:05:32 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
HTTP response body: {'status': 'error', 'code': 429, 'errors': ['Too many requests'], 'statuspage': 'http://status.us5.datadoghq.com', 'twitter': 'http://twitter.com/datadogops', 'email': 'support@datadoghq.com'}

To Reproduce
See description above

Expected behavior
I expect library to sleep for x-ratelimit-reset time, just like it's described in tests, which introduced this functionality. Right now I need to add a sleep between requests to API as a workaround

Screenshots
N/A - logs attached

Environment and Versions (please complete the following information):
client library version 2.23.0

Additional context
Add any other context about the problem here.