requests: HTTP/1.1 with Content-Length and streaming raw by pablogventura · Pull Request #1124 · micropython/micropython-lib

pablogventura · 2026-06-19T14:58:30Z

Summary

Send HTTP/1.1 requests (was HTTP/1.0).
Parse response status/headers via a small _http module.
Support Content-Length response bodies without buffering in request().
Preserve .raw as a live BodyStream wrapper; .content remains lazy.

Closes the approach in #1119 (which buffered the full body and closed the socket). Chunked responses, max_body, and relative redirects are intentionally left for follow-up PRs.

Testing

micropython test_requests.py (unix port) — 13 mock tests including incremental .raw.read().
CI: ruff, codespell, package tests.

Trade-offs and Alternatives

Chunked response bodies still raise ValueError (unchanged from master until a follow-up PR).
BodyStream adds a small wrapper object (~two fields) instead of buffering the body in request().
stream=True is not implemented yet (issue requests: stream parameter is noop #777).

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

Introduce read_status_line(), read_headers(), and BodyStream with Content-Length and until-close modes. No request() changes yet. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Wire _http into request(); send HTTP/1.1; return BodyStream as .raw without reading or closing the socket in request(). Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Adapt outbound tests to HTTP/1.1; add Content-Length and incremental .raw.read() coverage. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Document HTTP/1.1 requests and Content-Length body reads via .raw. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Read part of the body via .raw then the remainder via .content; avoid "hel" token flagged by codespell. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-19T15:07:42Z

ESP32 hardware testing

Tested on device connected via /dev/ttyACM0:


Board	ESP32
Firmware	MicroPython v1.29.0-preview.337 (2026-06-01)
WiFi	LosMius (STA), device IP `192.168.1.151`
Package deployed	`/lib/requests/__init__.py` + `/lib/requests/_http.py` from this PR

Mock tests (`test_requests.py`) — 13/13 OK

All existing mock tests pass on-device (outbound HTTP/1.1, Content-Length inbound, incremental .raw.read(), chunked response still raises ValueError).

Live tests (LAN HTTP server at `192.168.1.72:58080`) — 4/4 OK

Test	Result
`GET /get` with Content-Length body	200, 11 bytes
`GET /bytes/8` via `.content`	`abcdefgh`, socket closed after read
`GET /bytes/10` incremental `.raw.read(3)` + `.raw.read(3)` + `.raw.read()`	`3+3+4` bytes → `0123456789`
`GET /chunked`	`ValueError: Unsupported Transfer-Encoding: chunked` (expected for v1)

Server log confirmed outbound requests use HTTP/1.1:

GET /get HTTP/1.1
GET /bytes/8 HTTP/1.1
GET /bytes/10 HTTP/1.1
GET /chunked HTTP/1.1

Notes

httpbin.org returned 503 from this network, so live body tests used a local test server instead.
Mock tests replace sys.modules["socket"]; live tests reload usocket afterwards before hitting the network.

dpgeorge · 2026-06-21T13:35:00Z

@@ -1,5 +1,7 @@
 import socket

+from ._http import open_body, read_headers, read_status_line


I don't think these functions need to be in a separate module. MicroPython aims to be minimal, and we prefer minimal code over breaking things up into lots of modules.

So, please put everything from _http.py back in this file. That will also make it easier to review the changes.

Per review feedback: keep the package in a single module for easier review. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-21T20:22:11Z

Done — I moved everything from _http.py into __init__.py and dropped the extra module (commit 17f78a5).

Re-ran test_requests.py on the unix port locally (13/13). CI is green again on my side.

Let me know if anything else should change before you take another look.

pablogventura · 2026-06-21T20:25:31Z

Also re-tested on the same ESP32 as in my earlier comment (ESP32-D0WD on /dev/ttyACM0, MicroPython v1.29 preview, WiFi at home). After inlining _http.py into __init__.py, mock tests still pass on-device (test_requests.py, 13/13) and live HTTP over the LAN worked as before.

dpgeorge · 2026-06-22T01:34:46Z

+    return None
+
+
+def read_status_line(stream):


This function is used only once so please inline it at its point of use.

dpgeorge · 2026-06-22T01:34:56Z

+    return status, reason
+
+
+def read_headers(stream, parse_headers=True):


This function is used only once so please inline it at its point of use.

dpgeorge · 2026-06-22T01:35:03Z

+        self._sock.close()
+
+
+def open_body(stream, headers):


This function is used only once so please inline it at its point of use.

Fold read_status_line, read_headers, and open_body into request() per review feedback. Keep BodyStream as the only helper class. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-22T11:53:50Z

Thanks for the review, @dpgeorge — much appreciated.

I've inlined read_status_line, read_headers, and open_body directly into request() (commit 03a85e3). BodyStream remains as the only helper class. _header_get was folded into the body-setup block as well.

CI should re-run shortly. Happy to adjust anything else.

Use resp_d.items() when scanning for Location header. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-22T11:57:12Z

Re-tested after inlining the helpers and the ruff fix (commit d3c3628):

Unix port - micropython test_requests.py: 13/13 OK

ESP32 (ESP32-D0WD, /dev/ttyACM0, MicroPython v1.29 preview, WiFi at home):

Mock tests (test_requests.py): 13/13 OK
Live HTTP over LAN (local test server): 4/4 OK (Content-Length, incremental .raw.read(), chunked -> ValueError)

CI re-running on d3c3628.

dpgeorge · 2026-06-22T12:07:24Z

-        reason = ""
-        if len(l) > 2:
-            reason = l[2].rstrip()
+        line = s.readline()


Please keep original code unchanged if possible. In this case it's just renaming l to line, and that doesn't really need to change. It makes it much easier to review if the diff is minimal.

dpgeorge · 2026-06-22T12:08:11Z

-            reason = l[2].rstrip()
+        line = s.readline()
+        if not line:
+            raise ValueError("HTTP error: empty status line")


I don't think this check is needed. If line is empty then the len(parts) < 2 will catch that.

dpgeorge · 2026-06-22T12:08:30Z

+        line = s.readline()
+        if not line:
+            raise ValueError("HTTP error: empty status line")
+        parts = line.split(None, 2)


Please keep this as l = l.split(None, 2) to keep the diff minimal (it's also more efficient bytecode to reuse the variable).

dpgeorge · 2026-06-22T12:10:20Z

-            l = s.readline()
-            if not l or l == b"\r\n":
+            line = s.readline()
+            if not line or line == b"\r\n":


Please keep these 2 lines as they were.

dpgeorge · 2026-06-22T12:10:49Z

+        status = int(parts[1])
+        reason = parts[2].rstrip() if len(parts) > 2 else ""
+
+        resp_d = {}


This can be set to None and save a dict alloc if parse_headers is False. That's actually how the code was, so best to keep it as it was.

Keep l-based status/header parsing, resp_d = None when headers disabled, and in-loop Location/Transfer-Encoding checks. Only add BodyStream and HTTP/1.1 where needed. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-22T12:18:47Z

Thanks again for the detailed review, @dpgeorge.

Addressed the latest round in the new commit:

Restored l-based status/header parsing (no line/parts rename)
Removed the extra empty status-line check
Restored reason = "" / if len(l) > 2: as before
Restored resp_d = None when parse_headers is False
Restored in-loop Transfer-Encoding / Location handling from master
Kept only the minimal additions: HTTP/1.1, BodyStream, Content-Length limit

Re-tested on unix port: test_requests.py 13/13 OK.

Happy to adjust further if needed.

dpgeorge · 2026-06-22T12:27:49Z

+        remaining = None
+        if resp_d is not None:
+            for k, v in resp_d.items():
+                if k.lower() == "content-length":


To save code size, this check for content-length can probably go up in the code where resp_d is being populated.

dpgeorge · 2026-06-22T12:28:58Z

+                if k.lower() == "content-length":
+                    remaining = int(v)
+                    break
+        resp = Response(BodyStream(s, remaining))


If remaining is None I suggest not wrapping in BodyStream, it doesn't need it. So that means the behaviour is unchanged from before if there's no content-length, and the BodyStream class can be simpler (because remaining is now non-None).

Detect Content-Length while parsing headers; pass the socket through unchanged otherwise. Simplify BodyStream to require a byte count. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-22T12:44:40Z

Thanks for the follow-up comments, @dpgeorge.

Addressed in commit 8b20e5c:

Detect Content-Length while populating resp_d in the header loop (removed the extra scan at the end)
Use Response(s) when there is no Content-Length (unchanged behaviour from master)
Wrap in BodyStream only when Content-Length is present; BodyStream now always takes a byte count

Re-tested on unix port: test_requests.py 13/13 OK. CI green on 8b20e5c.

dpgeorge · 2026-06-22T12:54:42Z

+        data = self._sock.read(n)
+        self._remaining -= len(data)
+        if self._remaining > 0 and not data:
+            raise ValueError("Connection closed before Content-Length satisfied")


How does CPython's requests behave in this situation, does it raise an exception, or just return short data?

(Also, the check can be just if not data:.)

Use `if not data:` when a read returns empty before Content-Length is satisfied. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura · 2026-06-22T13:30:13Z

Good question. I checked CPython 3.12 with requests 2.31 / urllib3 2.0.7 on a response with Content-Length: 10 but only 3 body bytes before close:

response.raw.read(n) with partial data available: returns the short data (e.g. read(10) -> b"abc"), no exception on that call.
A later read once the socket is exhausted but bytes are still expected per Content-Length: urllib3 raises IncompleteRead (requests wraps it as ProtocolError / ChunkedEncodingError).
.content (full read): raises IncompleteRead.

So streaming behaviour is: partial reads are fine; exception when you try to read past EOF with Content-Length still outstanding. That matches what BodyStream does (ValueError instead of IncompleteRead, for simplicity).

Applied your suggestion in 6372c93: the check is now if not data:.
Re-tested on unix port: test_requests.py 13/13 OK.

pablogventura added 4 commits June 19, 2026 11:58

requests: Add _http helpers and BodyStream for Content-Length.

ba24eff

Introduce read_status_line(), read_headers(), and BodyStream with Content-Length and until-close modes. No request() changes yet. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

requests: Use HTTP/1.1 and preserve Response.raw streaming.

00ccbd2

Wire _http into request(); send HTTP/1.1; return BodyStream as .raw without reading or closing the socket in request(). Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

requests: Add tests for HTTP/1.1 and streaming raw.

a529d9e

Adapt outbound tests to HTTP/1.1; add Content-Length and incremental .raw.read() coverage. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

requests: Update README and bump version to 0.10.3.

4cd8564

Document HTTP/1.1 requests and Content-Length body reads via .raw. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

pablogventura mentioned this pull request Jun 19, 2026

requests: HTTP/1.1 response bodies and max_body limit #1119

Closed

requests: Fix incremental raw test and codespell false positive.

1205b33

Read part of the body via .raw then the remainder via .content; avoid "hel" token flagged by codespell. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

dpgeorge reviewed Jun 21, 2026

View reviewed changes

requests: Inline _http helpers into __init__.py.

17f78a5

Per review feedback: keep the package in a single module for easier review. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

dpgeorge reviewed Jun 22, 2026

View reviewed changes

requests: Inline single-use helpers in request().

03a85e3

Fold read_status_line, read_headers, and open_body into request() per review feedback. Keep BodyStream as the only helper class. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

requests: Fix ruff PLC0206 in redirect header lookup.

d3c3628

Use resp_d.items() when scanning for Location header. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

dpgeorge reviewed Jun 22, 2026

View reviewed changes

requests: Restore original parsing style per review.

2e23599

Keep l-based status/header parsing, resp_d = None when headers disabled, and in-loop Location/Transfer-Encoding checks. Only add BodyStream and HTTP/1.1 where needed. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

dpgeorge reviewed Jun 22, 2026

View reviewed changes

requests: Use BodyStream only when Content-Length is set.

8b20e5c

Detect Content-Length while parsing headers; pass the socket through unchanged otherwise. Simplify BodyStream to require a byte count. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

dpgeorge reviewed Jun 22, 2026

View reviewed changes

requests: Simplify BodyStream EOF check per review.

6372c93

Use `if not data:` when a read returns empty before Content-Length is satisfied. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

		@@ -1,5 +1,7 @@
		import socket

		from ._http import open_body, read_headers, read_status_line

		return status, reason


		def read_headers(stream, parse_headers=True):

Conversation

pablogventura commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Generative AI

Uh oh!

pablogventura commented Jun 19, 2026

ESP32 hardware testing

Mock tests (test_requests.py) — 13/13 OK

Live tests (LAN HTTP server at 192.168.1.72:58080) — 4/4 OK

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogventura commented Jun 21, 2026

Uh oh!

pablogventura commented Jun 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogventura commented Jun 22, 2026

Uh oh!

pablogventura commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogventura commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogventura commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogventura commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pablogventura commented Jun 19, 2026 •

edited

Loading

Mock tests (`test_requests.py`) — 13/13 OK

Live tests (LAN HTTP server at `192.168.1.72:58080`) — 4/4 OK

pablogventura commented Jun 22, 2026 •

edited

Loading