Add implementation of streaming abstractions - #535
Open
pjbull wants to merge 11 commits into
Open
Conversation
Contributor
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #535 +/- ##
========================================
+ Coverage 94.3% 94.8% +0.4%
========================================
Files 28 33 +5
Lines 2267 3077 +810
========================================
+ Hits 2140 2917 +777
- Misses 127 160 +33
🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds streaming I/O support to cloudpathlib, enabling direct file streaming from/to cloud storage (S3, Azure Blob Storage, GCS, and HTTP/HTTPS) without local caching. Key changes include:
- New
FileCacheMode.streamingenum value for direct streaming without caching CloudBufferedIOandCloudTextIOclasses implementing standard Python I/O interfaces- Provider-specific raw I/O adapters for efficient range requests and multipart uploads
Reviewed Changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 28 comments.
Show a summary per file
| File | Description |
|---|---|
cloudpathlib/enums.py |
Added streaming enum value to FileCacheMode |
cloudpathlib/cloud_io.py |
New module with CloudBufferedIO, CloudTextIO, and _CloudStorageRaw base classes |
cloudpathlib/cloudpath.py |
Updated CloudImplementation to track raw I/O classes; modified open() and __fspath__() for streaming mode |
cloudpathlib/client.py |
Added abstract methods for streaming I/O operations |
cloudpathlib/s3/s3_io.py |
S3-specific streaming I/O implementation |
cloudpathlib/azure/azure_io.py |
Azure-specific streaming I/O implementation |
cloudpathlib/gs/gs_io.py |
GCS-specific streaming I/O implementation |
cloudpathlib/http/http_io.py |
HTTP-specific streaming I/O implementation |
tests/test_cloud_io.py |
Comprehensive test suite for streaming I/O functionality |
tests/conftest.py |
Updated test fixtures to use CloudImplementation with raw I/O classes |
tests/test_cloudpath_instantiation.py |
Updated test to access dependencies_loaded via cloud_implementation |
tests/mock_clients/mock_s3.py |
Added mock methods for range requests and multipart uploads |
tests/mock_clients/mock_gs.py |
Added mock methods for range downloads and uploads |
tests/mock_clients/mock_azureblob.py |
Added mock methods for range downloads and block uploads |
tests/http_fixtures.py |
Added Range request handling to HTTP test server |
docs/docs/streaming_io.md |
Comprehensive documentation for streaming I/O feature |
docs/docs/caching.ipynb |
Updated caching documentation to include streaming mode |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| rig = CloudProviderTestRig( | ||
| path_class=LocalS3Path, | ||
| client_class=LocalS3Client, | ||
| cloud_implementation=local_s3_implementation, |
There was a problem hiding this comment.
Incorrect variable used: local_s3_implementation should be local_s3_cloud_implementation. This passes the wrong CloudImplementation object (the one from local.implementations.s3 module instead of the test-specific one created on line 559).
Suggested change
| cloud_implementation=local_s3_implementation, | |
| cloud_implementation=local_s3_cloud_implementation, |
|
|
||
| def validate_completeness(self) -> None: | ||
| expected = ["client_class", "path_class"] | ||
| expected = ["client_class", "path_class", "raw_io_class"] |
There was a problem hiding this comment.
The validation now requires raw_io_class to be present, but this is a breaking change. Existing custom implementations won't have this attribute set, causing them to fail validation. Consider making raw_io_class optional for backward compatibility, or only validate it when streaming mode is enabled.
Suggested change
| expected = ["client_class", "path_class", "raw_io_class"] | |
| expected = ["client_class", "path_class"] |
| # Flush the buffer first (which will call write() on the raw stream) | ||
| try: | ||
| self._buffer.flush() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| # Try to access the internal _closed attribute | ||
| if hasattr(self._buffer, "_closed"): | ||
| object.__setattr__(self._buffer, "_closed", True) | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| except Exception: | |
| except Exception: | |
| # Ignore errors when setting _closed; not critical for buffer cleanup. |
| path.client.file_cache_mode = original_mode | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| finally: | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| finally: | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| finally: | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| finally: | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| finally: | ||
| try: | ||
| path.unlink() | ||
| except Exception: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Introduces a new FileCacheMode.streaming that opens cloud files as
Python-standard BufferedIOBase/TextIOWrapper objects backed by provider
range-requests and multipart/resumable uploads, without requiring a local
cache copy.
New public API:
- CloudBufferedIO: BufferedIOBase wrapper for binary streaming
- CloudTextIO: TextIOWrapper for text streaming
- FileCacheMode.streaming enum value
Provider implementations:
- S3: multipart upload with 5 MiB minimum non-final part size;
_put_empty_object for zero-byte files
- GCS: blob.open("wb") resumable write stream; _put_empty_object
- Azure Blob: block-blob stage/commit pipeline; _put_empty_object
- HTTP(S): single-PUT via self.opener (honours SSL context)
- LocalClient: _LocalWriteStream + UUID-keyed multipart buffers for
concurrent-write safety
Correctness fixes included:
- _CloudStorageRaw.close() propagates finalize errors via try/finally
- seekable() returns False for write-only streams
- Exclusive-create ("x") mode checked against cloud existence
- Append/update modes ("a", "+") fall back to file-cache path
- Client streaming methods default to NotImplementedError (not
@AbstractMethod) so custom Client subclasses don't require
re-implementation
- HTTP _put_data uses self.opener for SSL-aware PUT requests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_s3_multipart_upload, test_azure_block_upload, test_gs_resumable_upload were skipping on ALL rigs because hasattr(rig, "s3_path") etc. is always False; switch to rig.path_class.cloud_prefix comparisons so they run on the correct rigs - gs/gsclient.py GCSNotFound fallback: type: ignore[misc, assignment] - azure/azblobclient.py: e.error_code → e.error.code (guarded for None) - gs/gs_io.py: annotate _writer as Optional[Any] to satisfy mypy Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Review fixes (verified by adversarial review of this PR): - write() advances the raw position so tell() is correct on streaming write streams (fixes corrupt output from zipfile and other position-dependent writers) - seek() on write-only raw streams raises io.UnsupportedOperation as documented instead of silently accepting the seek - GS range errors are matched structurally (status 416) instead of by a "416" substring that could swallow unrelated transient errors - Azure block IDs are namespaced by a per-upload session ID so concurrent writers cannot clobber each other's staged blocks - read() to EOF fetches the remainder in one ranged request (readall) instead of one request per 8 KiB - copy()/rename()/replace() work in streaming mode by streaming between clients instead of round-tripping through fspath - streaming writes honor force_overwrite_to_cloud, raising OverwriteNewerCloudError on close instead of clobbering a newer object - Client.__del__ cleans up streaming-mode cache files from the append/update fallback; docs no longer claim streaming never touches disk - streaming error paths raise cloudpathlib exception types; failed size lookups are memoized per stream; negative buffering matches builtins.open; behavior changes recorded in HISTORY.md Coverage and fidelity: - S3 extra-args filtering consults botocore's bundled service model instead of a hand-copied (and already stale) fallback table, with a drift-guard test - mocks now raise realistic SDK errors for missing objects and past-EOF ranges; HTTP test server clamps range ends per RFC 7233; GS mock enforces the 5 MiB non-final part minimum - default streaming buffer raised to 5 MiB; regression tests for all of the above plus parquet-over-streaming compatibility (pyarrow dev dep) Streaming concurrency: - new streaming_max_concurrency client parameter: bounded background part uploads while writing and read-ahead prefetch while reading, sequential by default - GS streaming writes use the XML API multipart upload (via the SDK's transfer-manager machinery) with in-memory parts, replacing the resumable write stream and unifying all multipart providers on one write contract; mock GS implements the XML MPU wire protocol Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pjbull
marked this pull request as ready for review
July 20, 2026 16:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.