requests: Add chunked response body support.#1135
Conversation
Decode Transfer-Encoding: chunked response bodies incrementally with a ChunkedStream wrapper, keeping .raw streaming (read/readinto) and .content lazy without buffering everything in request(). Chunk extensions are skipped and trailers are discarded. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Replace the test that expected a ValueError for chunked responses with coverage of decoding via .content and incremental .raw.readinto(). Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Note chunked response decoding in the feature list and drop the outdated limitation and follow-up entry. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
| self._sock.close() | ||
|
|
||
|
|
||
| class ChunkedStream: |
There was a problem hiding this comment.
This is a lot of extra code. Can you find a way to reduce the code size, maybe combine this class with BodyStream?
To test code size, run mpy-cross __init__.py and look at the size of the resulting __init__.mpy file. Making that file smaller is the goal.
There was a problem hiding this comment.
Thanks for the review. I merged chunked decoding into BodyStream with a chunked flag and inlined the chunk header parsing in readinto(), removing the separate ChunkedStream class (commit 4822856).
__init__.mpy sizes measured with mpy-cross against upstream/master (no chunked support):
| Version | __init__.mpy |
vs master |
|---|---|---|
| master (no chunked) | 2929 bytes | - |
this PR, separate ChunkedStream |
3413 bytes | +484 (+16.5%) |
this PR, merged into BodyStream |
3239 bytes | +310 (+10.6%) |
Merging the two classes saves 174 bytes (-5.1%) vs the separate-class version.
Unix tests pass. ESP32 (ESP32_GENERIC over WiFi) re-tested against httpbin chunked endpoints (.content on /stream/2, readinto on /stream-bytes/64).
Trailers are still discarded and chunk extensions are still ignored. The Content-Length code path is unchanged from master.
Combine ChunkedStream with BodyStream to reduce __init__.mpy size while keeping incremental read/readinto behaviour for chunked responses. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Summary
Follow-up to #1124, which added HTTP/1.1 with Content-Length streaming and
raised a ValueError on
Transfer-Encoding: chunkedresponses. This PR addsincremental decoding of chunked response bodies so
requestscan consume thecommon case of servers that stream without a Content-Length.
A small
ChunkedStreamwrapper decodes the chunk framing on the fly. It keepsthe existing behaviour intact:
.raw.read(n)and.raw.readinto(buf)stream the decoded body withoutbuffering everything in
request()(works formip)..contentstays lazy and materialises the full body on demand.BodyStream(Content-Length) is untouched; chunked responses selectChunkedStreaminstead.Chunk extensions after the size are skipped and trailers are discarded.
Testing
micropython python-ecosys/requests/test_requests.pypasses,including a replaced test that decodes a chunked response via
.contentanda new test that consumes it via
.raw.readinto().http://httpbin.org/stream/2(decoded via.content) andhttp://httpbin.org/stream-bytes/64(consumed via.raw.readinto()with a16-byte buffer). Both returned the expected bytes.
Trade-offs and Alternatives
.contentstill materialises the whole body, consistent with the existingcontract;
readinto()is the incremental path.Generative AI
I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.