Chunked responses in aiohttp result in an attempt to allocate a 4 gigabyte buffer… and maybe it doesn't actually work anyway?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- networking
Research direction
Start with ChunkedClientResponse and compare its handling with ClientResponse.read, including the text and json entry points. Review how asyncio.StreamReader.read and readexactly are used, then verify that chunked responses can consume multiple chunks without attempting an enormous allocation. Done means chunked transfers are decoded correctly and the reported failure no longer occurs.
Written by the indexing model from the issue text.
Description
Commit b95ca24 replaced calls to asyncio.StreamReader.read with calls to asyncio.StreamReader.readexactly. One important distinction between these two methods is that read allows a value of -1 to be passed, which results in it reading until EOF is reached, whereas readexactly requires (as its name implies) an exact size. This change breaks "chunked" transfers, as they don't supply a "Content-Length" header. When that header isn't present, aiohttp defaults to passing -1 to the read method, resulting in it attempting to allocate a buffer of 4,294,967,126 bytes.
Interestingly, ChunkedClientResponse.read correctly decodes the content length from the first line of the response data (which is consistent with the HTTP 1.1 spec), but ignores that value in favor of the -1 passed from ClientResponse.text or ClientResponse.json. This is handled in ClientResponse.read by checking specifically for a value of -1 and using the asyncio.StreamReader.read method, but not in ChunkedClientResponse.read.
Upon further inspection…
After looking closer at ChunkedClientResponse and the HTTP 1.1 spec, I think it just doesn't work correctly at all. A chunk-encoded response can require multiple reads from the response stream, each of a length determined by a value encoded in its first line. The way the class is written assumes that the first chunk's size is the size of the full data (chunked encoding is specifically intended for situations where the full size isn't known when the transfer starts) and there is only a single chunk.
So hey, I'm going to rewrite it and do a PR.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 1.1k
- Avg merge
- 7d 4h
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from micropython/micropython-lib
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
micropython/micropython-lib#1074 ·
-
needs-info
Difficulty 1/5 Under an hour Newbie friendliness 72/100
micropython/micropython-lib#943 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
micropython/micropython-lib#931 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 62/100
micropython/micropython-lib#795 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
micropython/micropython-lib#762 ·
All issues in micropython/micropython-lib
Similar issues
-
area: harness bug status: needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Human-Agent-Society/reef#625 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
learningequality/kolibri#15351 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Name consistency Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
eellak/triplestore#65 · 1 comment ·