Chunked responses in aiohttp result in an attempt to allocate a 4 gigabyte buffer… and maybe it doesn't actually work anyway?

ChunkedClientResponse から始め、text と json のエントリーポイントを含めて、その処理を ClientResponse.read と比較します。asyncio.StreamReader.read と readexactly がどのように使われているかを確認し、その後、chunked responses が巨大な割り当てを試みることなく複数の chunks を消費できることを検証します。chunked transfers が正しくデコードされ、報告された failure が発生しなくなれば完了です。

Commit b95ca24 replaced calls to asyncio.StreamReader.read with calls to asyncio.StreamReader.readexactly. One important distinction between these two methods is that read allows a value of -1 to be passed, which results in it reading until EOF is reached, whereas readexactly requires (as its name implies) an exact size. This change breaks "chunked" transfers, as they don't supply a "Content-Length" header. When that header isn't present, aiohttp defaults to passing -1 to the read method, resulting in it attempting to allocate a buffer of 4,294,967,126 bytes.

Interestingly, ChunkedClientResponse.read correctly decodes the content length from the first line of the response data (which is consistent with the HTTP 1.1 spec), but ignores that value in favor of the -1 passed from ClientResponse.text or ClientResponse.json. This is handled in ClientResponse.read by checking specifically for a value of -1 and using the asyncio.StreamReader.read method, but not in ChunkedClientResponse.read.

Upon further inspection…

After looking closer at ChunkedClientResponse and the HTTP 1.1 spec, I think it just doesn't work correctly at all. A chunk-encoded response can require multiple reads from the response stream, each of a length determined by a value encoded in its first line. The way the class is written assumes that the first chunk's size is the size of the full data (chunked encoding is specifically intended for situations where the full size isn't known when the transfer starts) and there is only a single chunk.

So hey, I'm going to rewrite it and do a PR.