microsoft/autogen

Auto Recovery Logic for chat completion clients on different types of server errors

Open

#3632 opened on Oct 3, 2024

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (58,033 stars) (8,759 forks)batch import
help wantedproj-extensionssize-medium

Description

What feature would you like to be added?

It would be nice instead if we can support a model client that auto-recovers from various host-related errors with a configurable logic such as retries.

Here is a good example of an apparently transient error:

openai.APIStatusError: Error code: 424 - {'error': {'message': 'Error occurred while processing image(s).', 'type': 'failed_dependency', 'param': None, 'code': None}}

Moreover, even for rate limit errors like below, it doesn't always retry, there is some inconsistency happening

openai.RateLimitError: Error code: 429 - {'error': {'code': '429', 'message': 'Rate limit is exceeded. Try again in 1 seconds.'}}

Why is this needed?

Currently, the implementation of the chat completion client for OpenAI fails to recover and crashes for a host of errors (except for retries that are rate-limit related where it auto-retries), however, a host of other errors randomly occur and that forces the applications built on top of the client to have their own logic for auto-recovery.

Contributor guide

Tech stack
python
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonOpenAI APIRetry strategiesAutoGen contribution guide
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Examine the existing client code in autogen/oai/client.py or similar to understand current retry logic for rate limits. Implement a retry mechanism that can be configured via parameters to handle 424 and 429 errors. Use a library like tenacity for configurable retries. Ensure the solution aligns with the existing error handling pattern.