Problem with proxy and streaming

Open
  • Is something wrong?
#810 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
networking

Research direction

Start with the provided AsyncClient configuration and run the example with and without proxy="http://localhost:8080" and verify=False. Compare when streamed chunks become available, then trace the proxy and streaming entry points involved. Done means the reported configuration preserves incremental streaming, with the behavior verified against the no-proxy case.

Written by the indexing model from the issue text.

Description

Originally opened by @markowanga on 2024-02-13 01:54:52 in encode/httpx

I try to process streaming (return customer in chat). I need to use proxy. I have problem that response does not streaming when using proxy (all responses returned after all processed, no effect of writing text)

import asyncio
from typing import Optional

from httpx import AsyncClient
from openai import AsyncStream, AsyncOpenAI
from openai.types.chat import ChatCompletionChunk


async def get_openai_stream_agenerator() -> AsyncStream[ChatCompletionChunk]:
    client = AsyncOpenAI(
        http_client=AsyncClient(
            # when I comment these two lines streaming is ok
            proxy="http://localhost:8080",  # I'm using mitmproxy with basic configuration
            verify=False,
        )
    )
    messages = [
        {"role": "system", "content": "Return details about asking person"},
        {"role": "user", "content": "Iga Świątek"},
    ]
    response: AsyncStream[ChatCompletionChunk] = await client.chat.completions.create(
        model='gpt-4-0613',
        messages=messages,
        stream=True,
    )  # type: ignore
    return response


def get_delta_argument(chunk: ChatCompletionChunk) -> Optional[str]:
    if len(chunk.choices) > 0:
        return chunk.dict()['choices'][0]['delta']['content']
    else:
        return None


async def get_response_generator() -> None:
    async for it in await get_openai_stream_agenerator():
        value = get_delta_argument(it)
        if value:
            print(value, end="")
    print()


if __name__ == '__main__':
    asyncio.run(get_response_generator())

OS: macOS
Python version: Python v3.11.7
Library version: openai 1.12.0, httpx 0.26.0

Dominant language
Python
Stars
1.6k
Forks
103
Avg merge
1d 19h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydantic/httpx2

All issues in pydantic/httpx2

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.