Feature Request: Add async support for Generative AI Inference client

Inizia esaminando la PR #835 e il client sincrono esistente di Generative AI Inference che usa requests. Confronta il client async proposto, i suoi 15 test unitari e 7 test di integrazione con il supporto richiesto per chat, streaming, embeddings, autenticazione e context manager. Il lavoro è completato quando le operazioni asincrone richieste sono coperte e le versioni di Python e i modelli indicati rimangono supportati.

Feature Request

Description

Add native async/await support for the OCI Generative AI Inference client to enable non-blocking concurrent requests in async applications.

Problem Statement

The current SDK uses synchronous HTTP requests via the requests library. This causes issues in async applications:

1. Event loop blocking: Sync calls block the event loop in FastAPI, async agents, and other async frameworks
2. Limited concurrency: Cannot efficiently make concurrent API calls
3. Performance bottleneck: Sequential requests are significantly slower than concurrent alternatives

Proposed Solution

Add an AsyncGenerativeAiInferenceClient class that:

• Uses aiohttp for true async HTTP requests
• Reuses the existing OCI Signer for authentication
• Provides async versions of all GenAI operations (chat, streaming, embeddings, etc.)
• Supports async context manager pattern

Example Usage

import asyncio
from oci.generative_ai_inference import AsyncGenerativeAiInferenceClient

async def main():
    async with AsyncGenerativeAiInferenceClient(config) as client:
        # Concurrent requests - 3x faster than sequential
        results = await asyncio.gather(
            client.chat(details1),
            client.chat(details2),
            client.chat(details3),
        )

asyncio.run(main())

Performance Impact

Testing shows 2-3.5x throughput improvement for concurrent workloads:

Use Cases

1. FastAPI/async web frameworks: Non-blocking GenAI calls in async endpoints
2. LangChain agents: Concurrent tool calls and chain execution
3. Batch processing: Parallel processing of multiple prompts
4. Real-time applications: Low-latency streaming responses

Implementation

A reference implementation is provided in PR #835 with:

• Full async client implementation
• 15 unit tests
• 7 integration tests
• Tested on Python 3.9, 3.12, 3.13, 3.14
• Tested with 6 different models

Related

• PR: #835