HTTP/3 support via OpenSSL 3.5#13186
Conversation
6901c2f to
0a42b65
Compare
841b545 to
f00cdf3
Compare
f00cdf3 to
0c99cfc
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates Apache Traffic Server’s HTTP/3 / QUIC support to work with upstream OpenSSL 3.5 by (1) adding a quiche-to-OpenSSL 3.5 QUIC TLS callback compatibility shim and (2) introducing an optional OpenSSL-native QUIC backend that is mutually exclusive with quiche. It also broadens and adds gold tests so core HTTP/3 scenarios can run across either QUIC backend where appropriate.
Changes:
- Add CMake feature detection + build options for OpenSSL 3.5 QUIC (native QUIC + QUIC TLS callbacks compatibility for quiche).
- Introduce an OpenSSL-native QUIC NetProcessor/NetVC/PacketHandler path and wire it into existing QUIC/HTTP3 plumbing.
- Expand/add gold tests and ATSReplay support for HTTP/3, plus minor HTTP/3 transaction / stream lifecycle fixes.
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gold_tests/timeout/quic_no_activity_timeout.test.py | Generalize QUIC gating to TS_USE_QUIC and keep quiche-specific assertions conditional. |
| tests/gold_tests/timeout/active_timeout.test.py | Run HTTP/3 active-timeout curl path under TS_USE_QUIC (still requires curl http3). |
| tests/gold_tests/headers/via.test.py | Run HTTP/3 portion of Via header test under TS_USE_QUIC. |
| tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml | New Proxy Verifier replay covering concurrent stream lifetime behavior. |
| tests/gold_tests/h3/replays/h3_sni.replay.yaml | Adjust SNI/authority values used by the HTTP/3 SNI replay. |
| tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml | New replay covering multi-connection HTTP/3 proxy verifier interop. |
| tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml | New replay exercising HTTP/3 active-timeout cleanup behavior. |
| tests/gold_tests/h3/h3_stream_lifetime.test.py | New ATSReplay-based test for stream lifetime handling. |
| tests/gold_tests/h3/h3_sni_check.test.py | Update QUIC gating + cert material for the SNI behavior test. |
| tests/gold_tests/h3/h3_session_ticket.test.py | New OpenSSL QUIC s_client-based session ticket test (OpenSSL 3.5+). |
| tests/gold_tests/h3/h3_proxy_verifier.test.py | New ATSReplay-based test for proxy verifier client interop. |
| tests/gold_tests/h3/h3_curl.test.py | New curl-based HTTP/3 interop test (requires curl http3 + --http3-only). |
| tests/gold_tests/h3/h3_active_timeout.test.py | New verifier-based HTTP/3 active-timeout cleanup test. |
| tests/gold_tests/autest-site/ats_replay.test.ext | Extend ATSReplay to pass http3_ports when QUIC is enabled in replay config. |
| src/traffic_server/traffic_server.cc | Switch QUIC include gate to TS_USE_QUIC. |
| src/traffic_layout/info.cc | Expose TS_HAS_OPENSSL_QUIC in feature reporting. |
| src/proxy/http3/test/test_Http3FrameDispatcher.cc | Adjust dispatcher test expectations for incremental feed behavior. |
| src/proxy/http3/Http3Transaction.cc | Improve stream-closure, timeout, and deferred read/write event handling; add safe self-delete gating. |
| src/proxy/http3/Http3StreamDataVIOAdaptor.cc | Fix reader lifecycle and make finalize() idempotent. |
| src/proxy/http3/Http3Session.cc | Avoid iterator invalidation while iterating transactions on session close. |
| src/proxy/http3/Http3HeaderVIOAdaptor.cc | Notify transaction when header decode completes (to schedule pending reads). |
| src/proxy/http3/Http3Frame.cc | Add DATA frame readiness parsing based on available payload bytes. |
| src/proxy/http3/Http3App.cc | Inform transaction on QUIC stream close for cleanup. |
| src/iocore/net/SSLSessionTicket.cc | Avoid hard-failing ticket callback when OpenSSL QUIC listener parses tickets before NetVC binding. |
| src/iocore/net/QUICNetVConnection.cc | Refactor stream send/recv to use QUICStreamIO interface and add stream I/O wrappers. |
| src/iocore/net/QUICMultiCertConfigLoader.cc | Add OpenSSL QUIC-specific SNI/cert selection + ALPN/curves hooks and TLS1.3-only enforcement for QUIC. |
| src/iocore/net/quic/QUICStreamVCAdapter.cc | Split read vs consume semantics and improve write reenable behavior. |
| src/iocore/net/quic/QUICStreamManager.cc | Implement local uni/bidi stream id allocation. |
| src/iocore/net/quic/QUICStreamAdapter.cc | Add consume() API and move on_read() notification there. |
| src/iocore/net/quic/QUICStream.cc | Abstract stream I/O via QUICStreamIO, support partial sends with pending blocks, and add on-write notification. |
| src/iocore/net/quic/QUICGlobals.cc | Adjust session handling compilation gates for new QUIC backends. |
| src/iocore/net/quic/QUICConfig.cc | Add QUIC server SSL_CTX creation via OpenSSL native QUIC method; gate quiche-only config APIs. |
| src/iocore/net/quic/OpenSSLQuicCompat.cc | New compatibility shim exporting quiche-expected legacy QUIC TLS callback symbols via OpenSSL 3.5 QUIC TLS callbacks API. |
| src/iocore/net/quic/CMakeLists.txt | Make quiche linkage conditional and add static compat shim library when required. |
| src/iocore/net/P_QUICPacketHandler.h | Add OpenSSL QUIC includes and OpenSSL-specific handler members/methods under feature guards. |
| src/iocore/net/P_QUICNetVConnection.h | Add OpenSSL QUIC init/stream plumbing, introduce QUICStreamIO, and add feature guards for quiche-only fields. |
| src/iocore/net/P_QUICNetProcessor.h | Make quiche-only members conditional and adjust signatures for new handler creation paths. |
| src/iocore/net/OpenSSLQUICPacketHandler.cc | New OpenSSL QUIC listener-based inbound packet handler implementation. |
| src/iocore/net/OpenSSLQUICNetVConnection.cc | New OpenSSL QUIC-based NetVC implementing stream accept/read/write via OpenSSL QUIC stream APIs. |
| src/iocore/net/OpenSSLQUICNetProcessor.cc | New OpenSSL QUIC NetProcessor wiring for inbound QUIC accept. |
| src/iocore/net/CMakeLists.txt | Select OpenSSL QUIC vs quiche QUIC sources and link dependencies based on feature flags. |
| include/tscore/ink_config.h.cmake.in | Add TS_HAS_OPENSSL_QUIC feature macro. |
| include/proxy/http3/Http3Transaction.h | Add stream close notification, new read scheduling helper, close-state tracking, and safe delete gating. |
| include/proxy/http3/Http3StreamDataVIOAdaptor.h | Track reader and finalized state for safe finalize/destruction. |
| include/proxy/http3/Http3HeaderVIOAdaptor.h | Minor formatting cleanup. |
| include/proxy/http3/Http3Frame.h | Allow DATA frame to override parsing readiness behavior. |
| include/iocore/net/QUICMultiCertConfigLoader.h | Add OpenSSL QUIC hook overrides behind TS_HAS_OPENSSL_QUIC. |
| include/iocore/net/quic/QUICStreamVCAdapter.h | Add _consume() override hook. |
| include/iocore/net/quic/QUICStreamManager.h | Make stream-creation methods virtual with out-params; track next local stream ids. |
| include/iocore/net/quic/QUICStreamAdapter.h | Add consume() API and require _consume() implementations. |
| include/iocore/net/quic/QUICStream.h | Introduce QUICStreamIO interface and pending-send tracking for partial writes. |
| include/iocore/net/quic/QUICConnection.h | Add optional on_stream_updated() hook. |
| include/iocore/net/quic/QUICConfig.h | Gate quiche-only APIs and declare quic_new_server_ssl_ctx(). |
| include/iocore/net/quic/Mock.h | Update mock stream adapter behavior to align with read/consume split. |
| CMakePresets.json | Add CI preset wiring for OpenSSL QUIC enablement and adjust Fedora preset inheritance. |
| CMakeLists.txt | Add ENABLE_OPENSSL_QUIC, OpenSSL QUIC feature checks, mutual exclusivity with quiche, and quiche static selection under compat mode. |
| cmake/Findquiche.cmake | Support selecting static quiche when required by compat shim linkage needs. |
| cmake/CheckOpenSSLHasQuicTlsCbs.cmake | New compile-check for OpenSSL QUIC TLS callback API availability. |
| cmake/CheckOpenSSLHasNativeQuic.cmake | New compile-check for OpenSSL native QUIC listener/stream APIs availability. |
0c99cfc to
1ecea18
Compare
1ecea18 to
7b26164
Compare
phongn
left a comment
There was a problem hiding this comment.
🤖 Automated review by Claude Code, posted on behalf of @phongn.
Inline comments below are from an offline analysis pass over the diff (correctness + security focus). They range from a couple of suggested fixes to confirmations and things worth verifying. Summary findings, in rough priority: (1) stream_write_capacity error path, (2) create_*_stream quiche behavior change to regression-test, (3) whole-DATA-frame buffering, (4) redundant SNI/cert callbacks, (5) a maintenance note on stream-map iteration, and (6) a suggestion to give OpenSSLQuicCompat.cc a dedicated security pass. Nice work overall — the QUICStreamIO abstraction and test coverage are solid.
| return std::make_unique<QUICConnectionError>(QUICTransErrorCode::INTERNAL_ERROR, "QUIC connection is not initialized"); | ||
| } | ||
|
|
||
| SSL *stream_ssl = SSL_new_stream(this->_ssl, flags | SSL_STREAM_FLAG_NO_BLOCK); |
There was a problem hiding this comment.
🤖 Automated review by Claude Code, posted on behalf of @phongn.
Minor: SSL_STREAM_FLAG_NO_BLOCK is already OR'd in by both callers (OpenSSLQUICStreamManager::create_uni_stream/create_bidi_stream), so OR-ing it again here is redundant. Separately, on SSL_new_stream failure this always reports STREAM_LIMIT_ERROR, even when the cause isn't a stream-limit condition.
|
[approve ci autest 2] |
7b26164 to
8da4a30
Compare
| UDPNetProcessorInternal::udp_read_from_net(UDPNetHandler *nh, UDPConnection *xuc) | ||
| { | ||
| #if HAVE_RECVMMSG | ||
| read_multiple_messages_from_net(nh, xuc); | ||
| while (read_multiple_messages_from_net(nh, xuc)) {} | ||
| #else |
8da4a30 to
c161a49
Compare
c161a49 to
006cd26
Compare
006cd26 to
a2e5c6e
Compare
a2e5c6e to
2342450
Compare
| void | ||
| UDPNetProcessorInternal::udp_read_from_net(UDPNetHandler *nh, UDPConnection *xuc) | ||
| { | ||
| #if HAVE_RECVMMSG | ||
| read_multiple_messages_from_net(nh, xuc); | ||
| while (read_multiple_messages_from_net(nh, xuc)) {} | ||
| #else |
2342450 to
952254e
Compare
# Overview This patch extends the HTTP/3 autest coverage, using curl, golang, and Proxy Verifier HTTP/3 clients to generate their implementations of h3 traffic. It also adds request and response bodies of various sizes, including "large" 300k bodies to exercise multiple packet, buffer, and flow control ATS HTTP/3 implementations. It also exercises some interesting requests, such as HEAD and 204 responses. This patch also includes the various production fixes needed for these tests. # Issues Found and their Fixes ## UDP batches could stall large H3 transfers Large request and response bodies exposed a UDP receive starvation bug in the UDP read path. On systems using `recvmmsg()` with edge-triggered readiness, ATS could read one full batch of datagrams and then leave the rest queued in the kernel without another readable event to wake the QUIC stack. This changes `UDPNetProcessorInternal::read_multiple_messages_from_net()` in `src/iocore/net/UnixUDPNet.cc` to return whether the kernel supplied a full batch. `udp_read_from_net()` now processes a bounded number of full batches per event, preserving UDP batching for H3 while avoiding both unread UDP bursts and unbounded net-thread monopolization under sustained QUIC load. ## QUIC stream writes consumed data before quiche accepted it The stream write path consumed the `QUICStreamVCAdapter` write reader inside `_read()`, before `QUICStream::send_data()` knew whether `quiche_conn_stream_send()` had accepted the bytes. When quiche accepted only a partial write or returned a flow-control error, ATS could lose stream data and report write progress too early. This makes `QUICStream::send_data()` keep a pending `IOBufferBlock`/FIN pair until quiche reports successful consumption, and only then calls the new `QUICStreamAdapter::consume()` hook. The concrete reader accounting lives in `QUICStreamVCAdapter::_consume()`, while `QUICStream::has_data_to_send()`, `QUICStream::on_write()`, and `QUICNetVConnection::on_stream_updated()` make newly writable stream data schedule packet writes again. This also treats completed finite writes with only FIN left as writable stream state, so empty bodies and fully consumed bodies still close the H3 stream cleanly. ## H3 transaction cleanup raced with stream closure The timeout and stream lifetime tests exposed cases where an `HQTransaction` could be deleted while an event handler was still active, or while the QUIC stream adapter still had read/write cleanup to finish. That left later stream-close and timeout paths touching state that had already been torn down. This adds explicit transaction lifetime state in `HQTransaction`: `_closed`, `_stream_closed`, `_event_handler_active`, and `_is_write_buffer_flushed()`. `Http3App::on_stream_close()` now calls `HQTransaction::stream_closed()` while holding the transaction mutex, and `HQTransaction::_delete_if_possible()` waits until the transaction is done, the stream is closed or no longer readable, and pending writes have flushed before deleting the transaction. ## H3 read completion could run before headers and DATA were settled The H3 request read path could signal completion before asynchronous QPACK header decode and buffered DATA delivery had finished updating the sink VIO. That showed up around HEAD, 204, and stream-close timing because the HTTP state machine needed a stable view of whether headers were decoded and whether a request body existed. This updates `Http3HeaderVIOAdaptor::_on_qpack_decode_complete()` to add the printed header length to the sink VIO and notify `Http3Transaction::on_header_decode_complete()`, which schedules the appropriate read event. `Http3StreamDataVIOAdaptor::finalize()` now uses a persistent reader, writes buffered DATA into the sink VIO exactly once, and updates `ndone`/`nbytes` consistently before the transaction is signaled. ## The QPACK static table had drifted from the standard table The HEAD, 204, and quic-go coverage exposed that ATS's static QPACK table was not the table used by external HTTP/3 implementations. The extra zstd entry and modified `accept-encoding` value in `src/proxy/http3/QPACK.cc` shifted later static indexes, so an externally encoded `:status 204` could decode as a different status. This restores the standard static table entries by using `accept-encoding: gzip, deflate, br` and removing the non-standard `content-encoding: zstd` entry. The new 204 cases in `tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml` and `tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml` cover this interoperability point with Proxy Verifier and quic-go.
Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS callback API, but quiche still links against the older quictls/BoringSSL symbols. ATS therefore could not use the system OpenSSL library for downstream HTTP/3 without dragging in a different TLS stack. This adds CMake detection for the OpenSSL callback API and provides a private compatibility layer that maps quiche's legacy hooks onto SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS resolves the shim symbols locally and links the final binaries against the system OpenSSL libraries. This also relaxes verifier-only HTTP/3 AuTest gates that do not execute curl, so those tests can run when ATS has QUIC support but the installed curl lacks HTTP/3.
OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a quiche-backed HTTP/3 listener. Operators who want to use the system OpenSSL QUIC stack needed a separate downstream backend without changing the existing quiche path or origin HTTP/3 scope. This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's native QUIC listener and stream APIs for downstream HTTP/3. This keeps the backend mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC, and shares ATS's existing HTTP/3 stream handling above the transport. This also installs native-QUIC TLS callbacks for ALPN and SNI certificate selection before ATS has a QUIC NetVC to bind. OpenSSL native QUIC does not make a selected SSL_CTX certificate active via SSL_set_SSL_CTX alone, so this applies the selected cert, key, and chain to the connection SSL. This also broadens client-side HTTP/3 tests to run with either backend and hardens transaction cleanup when OpenSSL closes stream state before ATS finishes teardown. This caches stream identifiers, declines listener-time QUIC tickets until a NetVC is bound, and adds focused H3 lifecycle and session-ticket coverage.
952254e to
ae8c052
Compare
First Commit: HTTP/3 via OpenSSL 3.5 + quiche
Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS
callback API, but quiche still links against the older
quictls/BoringSSL symbols. ATS therefore could not use the system
OpenSSL library for downstream HTTP/3 without dragging in a different
TLS stack.
This adds CMake detection for the OpenSSL callback API and provides a
private compatibility layer that maps quiche's legacy hooks onto
SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS
resolves the shim symbols locally and links the final binaries against
the system OpenSSL libraries.
This also relaxes verifier-only HTTP/3 AuTest gates that do not execute
curl, so those tests can run when ATS has QUIC support but the installed
curl lacks HTTP/3.
Second Commit: HTTP/3 via OpenSSL 3.5 QUIC
OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a
quiche-backed HTTP/3 listener. Operators who want to use the system
OpenSSL QUIC stack needed a separate downstream backend without changing
the existing quiche path or origin HTTP/3 scope.
This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's
native QUIC listener and stream APIs for downstream HTTP/3. This keeps
the backend mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC,
and shares ATS's existing HTTP/3 stream handling above the transport.
This also installs native-QUIC TLS callbacks for ALPN and SNI
certificate selection before ATS has a QUIC NetVC to bind. OpenSSL
native QUIC does not make a selected SSL_CTX certificate active via
SSL_set_SSL_CTX alone, so this applies the selected cert, key, and chain
to the connection SSL.
This also broadens client-side HTTP/3 tests to run with either backend
and hardens transaction cleanup when OpenSSL closes stream state before
ATS finishes teardown. This caches stream identifiers, declines
listener-time QUIC tickets until a NetVC is bound, and adds focused H3
lifecycle and session-ticket coverage.
Testing
Production Testing
This is currently running in docs for https://docs.trafficserver.apache.org/ targets via Alt-Svc header rewrite rules to negotiate
h3if it is supported.CI AuTests/Fedora 44 Note
This patch is built for openssl 3.5. It assumes CI is running the autests in a fedora:44 container which has that as the system openssl. The autests will fail to build as such until we switch over CI to fedora:44, which is hopefully soon.