Retry mechanism behaves incorrectly when HTTP 429 is returned by Datadog

Bắt đầu bằng cách theo dõi cách tùy chọn enable_retry xử lý các phản hồi HTTP 429 trong client, sử dụng chuỗi lấy dashboard và các bài kiểm thử liên quan đến việc thử lại được mô tả trong issue. Tái hiện trường hợp bị giới hạn tốc độ và xác minh rằng client chờ đến x-ratelimit-reset trước khi thử lại, thay vì thoát ngay khi nhận 429 đầu tiên.

Describe the bug
Summary:
Script which fetches all dashboards in the loop exits with an error, when rate limit runs out, even though enable_retry option is turned on. According to debug logs, script exits on first HTTP 429 returned, with no retry attempted.

Details:
We have a script, which runs every night to fetch all dashboards from Datadog. It does it by fetching list of dashboards, and then goes one after another to fetch details of each one. After number of our dashboard grew, we have run into HTTP 429 errors due to Datadog's rate limit.

We decided to use retry option, which is built into the library since 2.16.0, but it seems it's not ready to how Datadog responds in case of hitting rate limit.

When I started the script in a loop with debug option enabled I see that Datadog returns HTTP 200 up until the moment when rate limit is reached, then next request gets HTTP 429 (API Keys removed from logs below):

# normal request before rate limit runs out
send: b'GET /api/v1/dashboard/<id-of-dashboard-59> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 1
header: x-ratelimit-reset: 29
header: x-ratelimit-name: dashboards_get_custom_api

# normal request, last one within the limits
send: b'GET /api/v1/dashboard/<id-of-dashboard-60> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 200 OK'
...
header: content-encoding: gzip
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# next request, this one gets HTTP 429
send: b'GET /api/v1/dashboard/<id-of-dashboard-61> Host: us5.datadoghq.com Accept-Encoding: gzip User-Agent: datadog-api-client-python/2.23.0
reply: 'HTTP/1.1 429 Too Many Requests'
...
header: x-ratelimit-limit: 60
header: x-ratelimit-period: 60
header: x-ratelimit-remaining: 0
header: x-ratelimit-reset: 28
header: x-ratelimit-name: dashboards_get_custom_api

# and at this point script fails with
Error: (429)
Reason: Too Many Requests
HTTP response headers: {'x-ratelimit-limit': '60', 'x-ratelimit-period': '60', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '28', 'x-ratelimit-name': 'dashboards_get_custom_api', 'content-type': 'application/json', 'Content-Length': '183', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'date': 'Mon, 25 Mar 2024 09:05:32 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
HTTP response body: {'status': 'error', 'code': 429, 'errors': ['Too many requests'], 'statuspage': 'http://status.us5.datadoghq.com', 'twitter': 'http://twitter.com/datadogops', 'email': 'support@datadoghq.com'}

To Reproduce
See description above

Expected behavior
I expect library to sleep for x-ratelimit-reset time, just like it's described in tests, which introduced this functionality. Right now I need to add a sleep between requests to API as a workaround

Screenshots
N/A - logs attached

Environment and Versions (please complete the following information):
client library version 2.23.0

Additional context
Add any other context about the problem here.