Streaming decompressor: suspend at symbol boundaries, not only block boundaries - #6
Open
alexey-milovidov wants to merge 11 commits into
Open
Streaming decompressor: suspend at symbol boundaries, not only block boundaries#6alexey-milovidov wants to merge 11 commits into
alexey-milovidov wants to merge 11 commits into
Conversation
libdeflate's compression API is one-shot: every call emits a complete, terminated DEFLATE stream. This makes it impossible to back a streaming compressor (HTTP responses, .gz files of unbounded size) without either buffering the whole input or emitting a multi-member stream, which some decoders read only partially. Add libdeflate_deflate_compress_stream_chunk(), which compresses a chunk as non-final, byte-aligned DEFLATE blocks (ending with an empty stored "sync flush" block, like zlib's Z_SYNC_FLUSH). The outputs of consecutive calls can be concatenated and terminated with a final block, producing a single valid DEFLATE/gzip/zlib member while keeping memory bounded to one chunk. Implementation: - A `stream_chunk` flag on the compressor makes deflate_flush_block() force every block non-final. Only the BFINAL bit value changes, not its cost, so the output-size accounting is unaffected. - Small inputs (<= max_passthrough_size, and level 0) are emitted as non-final stored blocks. - A sync-flush helper byte-aligns the output after compression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add libdeflate_deflate_compress_stream_chunk() for streaming compression
libdeflate's decompressor is one-shot: it decodes until BFINAL and cannot
resume on a partial stream, so it can't back a streaming decompressor for
arbitrarily large zlib/gzip/deflate input without buffering the whole thing.
Add a resumable variant that suspends at DEFLATE block boundaries. The only
cross-block state is {bitbuf, bitsleft} plus the 32 KiB history window, so
suspension is cheap: at each block boundary we checkpoint the byte-aligned
bit state, and if the input runs out (REFILL would overread and
end_of_input is false) or the output fills mid-block, we roll back to that
checkpoint and return STREAM_NEED_INPUT / STREAM_NEED_OUTPUT. The caller
supplies more input or drains output and calls again. Back-references are
resolved by having the caller place up to 32 KiB of previously produced
output immediately before 'out' and pass 'window_nbytes'; the match-offset
validation base is shifted accordingly.
The streaming code is a second instantiation of decompress_template.h guarded
by DEFLATE_STREAMING, so the existing one-shot decoder is byte-for-byte
unchanged (the only shared-macro change is a no-op OVERREAD_HANDLER() hook in
REFILL_BITS). A single block larger than the output buffer is handled by the
caller growing the buffer and retrying; mid-block suspension is intentionally
not supported.
Verified with 50000 randomized round-trip cases against zlib (random data
styles, sizes up to 3 MB, all levels, random input-chunk and output-region
sizes exercising both suspension paths and the window carry-over).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add libdeflate_deflate_decompress_stream() for streaming decompression
The one-shot decompressor selects a BMI2-optimized instantiation at runtime (x86/decompress_impl.h), but libdeflate_deflate_decompress_stream() called a single non-dispatched generic instantiation, so on BMI2-capable x86 the streaming path was stuck on the baseline code. Move the streaming instantiation into its own section after the one-shot dispatch (so DEFLATE_STREAMING doesn't leak into the one-shot BMI2 variant), add x86/decompress_stream_impl.h mirroring x86/decompress_impl.h, and route the public streaming entry through a dispatched function pointer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give the streaming decompressor the x86 BMI2 runtime dispatch
The crc32/adler32/decompress/stream-decompress dispatchers cache the runtime-resolved best implementation in a function pointer on the first call. The pointer was 'volatile', so the benign first-call race (every thread resolves the same pointer, a pure function of the CPU) was still undefined behavior and was flagged by ThreadSanitizer. Access the pointer with relaxed atomics (__atomic_load_n/__atomic_store_n) instead. The generated code is unchanged on real hardware, but the access is now well-defined and TSan-clean. Relaxed ordering is sufficient because no other memory is published through the pointer.
Make the lazy runtime CPU-feature dispatch thread-safe (relaxed atomics instead of volatile) so ThreadSanitizer no longer flags the benign first-call race in crc32/adler32/decompress/stream-decompress.
The previous fix (PR #4) made the per-codec dispatch function pointers (crc32_impl, adler32_impl, ...) atomic, but each dispatcher still calls get_x86_cpu_features() / get_arm_cpu_features(), which lazily initialize the global libdeflate_x86_cpu_features / libdeflate_arm_cpu_features bitmask on the first call. That global was only 'volatile', so a plain load racing with the store in libdeflate_init_*_cpu_features() is undefined behavior and was still flagged by ThreadSanitizer: WARNING: ThreadSanitizer: data race Write ... libdeflate_init_x86_cpu_features cpu_features.c Read ... get_x86_cpu_features cpu_features.h Location is global 'libdeflate_x86_cpu_features' This reproduced as a "Server died" failure when two HTTP connections first compressed a gzip response concurrently under TSan. Access the global with relaxed __atomic_load_n / __atomic_store_n, matching the dispatch-pointer fix. The first-call initialization is a benign race (every thread computes the same bitmask, a pure function of the CPU), and relaxed ordering suffices because no other memory is published through it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…boundaries The streaming decompressor used to keep its resume checkpoint only at DEFLATE block boundaries: on input exhaustion or output-buffer overflow it rolled back to the start of the current block and reported everything after it as unconsumed. For streams whose single block spans the whole input - the shape zlib-ng's deflate_quick path (compression level 1, the default of the official .NET SDK) emits - every refill re-decoded the block from its start, making streaming decompression quadratic in the compressed block size and forcing the caller to buffer the whole block. Now the checkpoint is re-taken at every symbol boundary in the generic decode loop, and two new decompressor fields (in_block, block_is_final) let a resumed call jump straight back into symbol decoding: the litlen/offset decode tables already persist in the decompressor between calls. The fastloop needs no checkpointing (and thus stays untouched): suspensions can only trigger from the generic loop, which always runs between the fastloop and input exhaustion. A suspension now loses at most one partially decoded symbol, so decompression is linear-time and constant-leftover regardless of the block structure. Also checkpoint-guard the case where implicit appended zero bytes were consumed (truncated data declared complete): that now fails with BAD_DATA at the checkpoint instead of producing an inconsistent input position. Verified with a standalone harness: 5000 randomized round-trips (zlib streams at all levels with random flush points, plus crafted static- and dynamic-Huffman single-block streams with matches reaching the full 32 KiB window, random input chunking down to 1 byte, random output regions), a progress contract check on every suspension, and zlib as decode oracle. Streaming throughput on normal multi-block streams is unchanged (685 vs 697 MB/s before/after on a 256 MB zlib-level-6 stream). ClickHouse/ClickHouse#114045
…ing before the first block header
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The streaming decompressor kept its resume checkpoint only at DEFLATE block boundaries: on input exhaustion or output-buffer overflow it rolled back to the start of the current block and reported everything after it as unconsumed. For streams whose single block spans the whole input — the shape zlib-ng's
deflate_quickpath (compression level 1, the default of the official .NET SDK) emits — every refill re-decoded the block from its start, making streaming decompression quadratic in the compressed block size and forcing the caller to buffer the whole block. This is the root cause of ClickHouse/ClickHouse#114045 (a 22 MB gzip HTTP body that took 0.3 s to ingest on ClickHouse 26.6 took 15 s on 26.7).Now the checkpoint is re-taken at every symbol boundary in the generic decode loop, and two new decompressor fields (
in_block,block_is_final) let a resumed call jump straight back into symbol decoding: the litlen/offset decode tables already persist in the decompressor between calls. The fastloop needs no checkpointing (and stays untouched): suspensions can only trigger from the generic loop, which always runs between the fastloop and input exhaustion. A suspension now loses at most one partially decoded symbol, so decompression is linear-time with O(1) leftover regardless of the block structure.Also, the checkpoints now guard the case where implicit appended zero bytes were consumed (truncated data declared complete via
end_of_input): that fails withLIBDEFLATE_BAD_DATAat the checkpoint instead of producing an inconsistent input position.Verified with a standalone harness: 5000+ randomized round-trips (zlib streams at all levels with random flush points, plus crafted static- and dynamic-Huffman single-block streams with matches reaching the full 32 KiB window, input chunking down to 1 byte, random output regions), a progress-contract check on every suspension, zlib inflate as decode oracle, and the one-shot decoder as cross-check. Streaming throughput on normal multi-block streams is unchanged (685 vs 697 MB/s before/after on a 256 MB zlib-level-6 stream, x86-64).
Related: ClickHouse/ClickHouse#114045
🤖 Generated with Claude Code