Retry mechanism behaves incorrectly when HTTP 429 is returned by Datadog

Comienza rastreando cómo la opción enable_retry gestiona las respuestas HTTP 429 en el cliente, usando la secuencia de obtención del dashboard y las pruebas relacionadas con los reintentos descritas en el issue. Reproduce el caso de límite de tasa y verifica que el cliente espera a x-ratelimit-reset antes de volver a intentarlo, en lugar de salir al recibir el primer 429.

Describe the bug
Summary:
Script which fetches all dashboards in the loop exits with an error, when rate limit runs out, even though enable_retry option is turned on. According to debug logs, script exits on first HTTP 429 returned, with no retry attempted.

Details:
We have a script, which runs every night to fetch all dashboards from Datadog. It does it by fetching list of dashboards, and then goes one after another to fetch details of each one. After number of our dashboard grew, we have run into HTTP 429 errors due to Datadog's rate limit.

We decided to use retry option, which is built into the library since 2.16.0, but it seems it's not ready to how Datadog responds in case of hitting rate limit.

When I started the script in a loop with debug option enabled I see that Datadog returns HTTP 200 up until the moment when rate limit is reached, then next request gets HTTP 429 (API Keys removed from logs below):

# normal request before rate limit runs out
send: b'GET /api/v1/dashboard/<id-of-dashboard-59> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 1
header: x-ratelimit-reset: 29
header: x-ratelimit-name: dashboards_get_custom_api

# normal request, last one within the limits
send: b'GET /api/v1/dashboard/<id-of-dashboard-60> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# next request, this one gets HTTP 429
send: b'GET /api/v1/dashboard/<id-of-dashboard-61> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 429 Too Many Requests'
...
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# and at this point script fails with
Error: (429)
Reason: Too Many Requests
HTTP response headers: {'x-ratelimit-limit': '60', 'x-ratelimit-period': '60', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '28', 'x-ratelimit-name': 'dashboards_get_custom_api', 'content-type': 'application/json', 'Content-Length': '183', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'date': 'Mon, 25 Mar 2024 09:05:32 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
HTTP response body: {'status': 'error', 'code': 429, 'errors': ['Too many requests'], 'statuspage': 'http://status.us5.datadoghq.com', 'twitter': 'http://twitter.com/datadogops', 'email': 'support@datadoghq.com'}

To Reproduce
See description above

Expected behavior
I expect library to sleep for x-ratelimit-reset time, just like it's described in tests, which introduced this functionality. Right now I need to add a sleep between requests to API as a workaround

Screenshots
N/A - logs attached

Environment and Versions (please complete the following information):
client library version 2.23.0

Additional context
Add any other context about the problem here.