TLS: in-memory transport does not deliver server records emitted at handshake completion

[mvandeberg]

TLS: in-memory transport does not deliver server records emitted at handshake completion

Labels: tls, bug, transport
Component: src/openssl (memory-BIO pair), src/wolfssl (custom I/O callbacks)
Blocks: tasks/tls/ocsp-stapling.md, tasks/tls/session-resumption.md

Summary

corosio does not hand the TLS library a socket. Each stream drives the SSL
state machine over an in-memory transport and shuttles bytes to the
underlying stream itself:

• OpenSSL: an OpenSSL memory BIO pair (BIO_new_bio_pair); the coroutine
  moves bytes between ext_bio_ and the underlying stream.
• WolfSSL: custom I/O callbacks (wolfSSL_SSLSetIORecv / SetIOSend)
  returning WOLFSSL_CBIO_ERR_WANT_READ/WANT_WRITE; the coroutine does the
  async read_some / write_some.

Records the server emits at or after the tail of the handshake — the OCSP
CertificateStatus staple, the TLS 1.2 NewSessionTicket, and TLS 1.3
post-handshake messages such as NewSessionTicket — are not observable on the
client through this transport, even though the identical library
configuration delivers them over a real socket. This is the shared root cause
that blocks OCSP stapling and session resumption.

Why this happens

The handshake coroutine returns as soon as the library reports the handshake
complete, having only flushed pending output. It never performs a trailing
read to pump records the peer sends after that point.

OpenSSL (src/openssl/src/openssl_stream.cpp, do_handshake, ~line 903):

if (ret == 1) {                 // SSL_connect / SSL_accept succeeded
    used_ = true;
    capture_alpn();
    ec = co_await flush_output();   // flush only — no final read_input()
    co_return {ec};
}

WolfSSL (src/wolfssl/src/wolfssl_stream.cpp, do_handshake, ~line 1085):
the success branch flushes any remaining output and breaks out of the loop
with no trailing read, the same shape.

Consequences:

• TLS 1.3 NewSessionTicket is a post-handshake record — it legitimately
  arrives only when the client next reads. Because the handshake returns
  without that read (and nothing pumps it until an application read that may
  never come), the ticket is never processed and the session is not
  resumable.
• TLS 1.2 NewSessionTicket / OCSP CertificateStatus are part of the
  server's handshake flight, yet were also found unobservable through the
  in-memory path — pointing at the byte-shuttle/drain logic not consuming the
  server's final records before the handshake is declared done.

Key evidence — isolate it from corosio's coroutine layer

The split was reproduced at the raw library level with no corosio code:
two SSL objects driven directly, once over socketpair(AF_UNIX, SOCK_STREAM) (works) and once over a hand-shuttled BIO_new_bio_pair
(fails). This localizes the fault to how the memory-BIO/callback transport is
pumped, not to corosio's coroutine machinery — and corosio's
flush_output / read_input mirror that same pump pattern.

A correctly driven BIO pair can deliver these records (many applications use
BIO pairs successfully), so the defect is a missing/early-terminated drain,
not a fundamental BIO-pair limitation.

How to confirm

Minimal standalone probe (no corosio), compiled against the same OpenSSL/
WolfSSL corosio links. Pick one observable record type:

• OCSP: server stapling on, client
  SSL_get_tlsext_status_ocsp_resp(client, &resp) → positive length over
  socketpair, -1 over the BIO pair.
• Session (TLS 1.2): SSL_session_reused(client) on a second connection
  → 1 over socketpair, 0 over the BIO pair.

Structure:

1. Baseline: connect the two SSL objects with SSL_set_fd over a
   socketpair; drive to completion; observe the record present.
2. Reproduction: wire each SSL to a BIO_new_bio_pair and shuttle bytes
   between the external BIOs by hand (mirror flush_output / read_input);
   observe the record absent.
3. Vary the shuttle: after both SSL_connect/SSL_accept report success,
   keep pumping (drain each side's BIO_ctrl_pending into the other, and let
   the client do one more SSL_read/SSL_peek) and check whether the record
   appears — this both confirms the mechanism and prototypes the fix.

Way forward

1. Root-cause with step (3) above: determine exactly which records are
   left unpumped and when (handshake-flight vs post-handshake), for TLS 1.2
   and TLS 1.3, on both backends.
2. Fix the drain. Likely a bounded, non-blocking post-success pump: after
   the library reports completion, keep shuttling while either side has
   pending output, and ensure a client-side read is available to process
   post-handshake records (TLS 1.3 tickets). It must be bounded — never
   block waiting for data that may never arrive; only drain what is already
   pending / already buffered by the peer.
3. Regression-guard the transport directly (a corosio-level test that
   asserts a post-handshake server record is observed by the client), so the
   two dependent features can rely on it.
4. Unblock and re-land tasks/tls/ocsp-stapling.md and
   tasks/tls/session-resumption.md.

Risks / considerations

• The pump must not deadlock or hang: it can only consume records the peer has
  already sent. A naive "read until ticket" loop would block on a peer that
  sends nothing.
• Interaction with the app-data read/write paths (read_input /
  flush_output are shared) — a post-handshake pump must not consume or
  reorder application data.
• TLS 1.3 tickets may arrive across several early reads; capturing them may
  additionally need SSL_CTX_sess_set_new_cb (noted in the session issue).
• Behavior must stay consistent across both transports (OpenSSL BIO pair and
  WolfSSL callbacks).

Acceptance criteria

• Root cause documented: which records, which TLS version, which backend.
• Client observes the server's OCSP staple and (TLS 1.2) resumable
  session through the in-memory transport, matching socket behavior.
• The pump is bounded — no new hang/deadlock; app data is never consumed
  or reordered by it.
• Transport-level regression test added; verified on OpenSSL and WolfSSL.

Pointers

• src/openssl/src/openssl_stream.cpp:41 — BIO data-flow banner
• src/openssl/src/openssl_stream.cpp — do_handshake (~863; ret == 1
  branch ~903 is the suspect), flush_output (~668), read_input (~699),
  init_ssl / BIO_new_bio_pair
• src/wolfssl/src/wolfssl_stream.cpp:42 — I/O data-flow banner; do_handshake
  (~1050; success branch ~1085), recv_callback / send_callback, and
  wolfSSL_SSLSetIORecv / SetIOSend setup (~1350)