microsoft/autogen

Auto Recovery Logic for chat completion clients on different types of server errors

Open

#3,632 opened on Oct 3, 2024

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (8,759 forks)batch import
help wantedproj-extensionssize-medium

Repository metrics

Stars
 (58,033 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

What feature would you like to be added?

It would be nice instead if we can support a model client that auto-recovers from various host-related errors with a configurable logic such as retries.

Here is a good example of an apparently transient error:

openai.APIStatusError: Error code: 424 - {'error': {'message': 'Error occurred while processing image(s).', 'type': 'failed_dependency', 'param': None, 'code': None}}

Moreover, even for rate limit errors like below, it doesn't always retry, there is some inconsistency happening

openai.RateLimitError: Error code: 429 - {'error': {'code': '429', 'message': 'Rate limit is exceeded. Try again in 1 seconds.'}}

Why is this needed?

Currently, the implementation of the chat completion client for OpenAI fails to recover and crashes for a host of errors (except for retries that are rate-limit related where it auto-retries), however, a host of other errors randomly occur and that forces the applications built on top of the client to have their own logic for auto-recovery.

Contributor guide

Research direction
Investigate the current error handling in the chat completion client and propose a retry logic for transient server errors, such as 424 and 429 status codes, with configurable retry limits and backoff strategies.
Tech stack
python
Domain
aibackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonOpenAI API experience
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40