Skip to content
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
21e66a3
add AsyncAbstractObjectStream
chandra-siri Sep 17, 2025
39503f4
keep _AsyncAbstractObjectStream private
chandra-siri Sep 17, 2025
a161fd0
Add _AsyncReadObjectStream and it's stubs
chandra-siri Sep 17, 2025
dd862a2
complete __init__ for read_obj_str
chandra-siri Sep 17, 2025
aaabfd7
remove unuseful comments
chandra-siri Sep 17, 2025
23eea96
add methods open close send recv
chandra-siri Sep 17, 2025
71a7a79
change read_handle type from 'str' to 'bytes'
chandra-siri Sep 18, 2025
827aec0
feat: add async_multi_range_downloader
chandra-siri Sep 18, 2025
5be7469
fix: read_ranges should have buffer as well
chandra-siri Sep 18, 2025
b3ad551
rename MultiRangeDownloader to AsyncMultiRangeDownloader
chandra-siri Sep 18, 2025
a87f2be
feat: implement download_ranges method
chandra-siri Sep 18, 2025
c2e3c7b
add BytesIO in doc string
chandra-siri Sep 18, 2025
3c0fd66
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 18, 2025
0810afc
fix doc strings, add licence and type hints
chandra-siri Sep 18, 2025
d43d889
Merge branch 'bidi_reads_1_abs_obj_stream' of github.com:googleapis/p…
chandra-siri Sep 18, 2025
a14bc68
pass abstract methods
chandra-siri Sep 18, 2025
fbacbb4
Merge branch 'bidi_reads_1_abs_obj_stream' of github.com:googleapis/p…
chandra-siri Sep 18, 2025
fd37489
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 18, 2025
635ad07
add handle param
chandra-siri Sep 18, 2025
ba453d4
include handle in tests
chandra-siri Sep 18, 2025
800c6df
remove unit tests for abstract class
chandra-siri Sep 18, 2025
8b40812
Merge branch 'bidi_reads_1_abs_obj_stream' of github.com:googleapis/p…
chandra-siri Sep 18, 2025
18529ad
edit doc string for _AsyncReadObjectStream
chandra-siri Sep 18, 2025
df2532e
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 18, 2025
b4da1ac
refactor unit tests for async_read_object_stream
chandra-siri Sep 18, 2025
6dec6c6
bucket_name and object_name cannot be NONE
chandra-siri Sep 18, 2025
90a65f6
Merge branch 'bidi_reads_1_abs_obj_stream' of github.com:googleapis/p…
chandra-siri Sep 18, 2025
a154905
bucket_name and object_name cannot be None
chandra-siri Sep 18, 2025
d69cd63
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 18, 2025
20afca4
Merge branch 'bidi_reads_3_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
14a2aba
simplyfy tests for open
chandra-siri Sep 19, 2025
078afca
simply tests for send recv and close
chandra-siri Sep 19, 2025
2054989
minor edit - add bidi-stream in doc string
chandra-siri Sep 19, 2025
2b9ae2e
Merge branch 'bidi_reads_1_abs_obj_stream' of github.com:googleapis/p…
chandra-siri Sep 19, 2025
c06896c
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
916e7f1
Merge branch 'bidi_reads_3_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
3658502
Merge branch 'bidi_reads_4_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
8366f0b
simplify unit tests
chandra-siri Sep 19, 2025
4a774ec
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
0991383
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 19, 2025
f589b89
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
06b102c
improve doc string
chandra-siri Sep 19, 2025
52494b4
fix unit tess in MRD
chandra-siri Sep 19, 2025
8e00701
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
0e60694
add checks for invalid inputs
chandra-siri Sep 19, 2025
961def8
Merge branch 'bidi_reads_2_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 19, 2025
e9d0f9e
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 22, 2025
c4f61ea
Merge branch 'bidi_reads_3_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 22, 2025
ef5d917
Merge branch 'bidi_reads_4_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 22, 2025
7fbf28c
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 22, 2025
dcb6a55
remove duplicated import
chandra-siri Sep 23, 2025
90d8597
remove unused import
chandra-siri Sep 23, 2025
229887b
Merge branch 'bidi_reads_3_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 23, 2025
5a6ffc7
Merge branch 'bidi_reads_4_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 23, 2025
9f15551
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 23, 2025
234336f
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 23, 2025
521154c
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 23, 2025
8338ab2
fix unit test
chandra-siri Sep 23, 2025
c3cb076
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 23, 2025
ac8e651
remove unused import
chandra-siri Sep 23, 2025
c6b1098
Merge branch 'bidi_reads_5_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 23, 2025
6469063
implement basic functionality for download_ranges
chandra-siri Sep 24, 2025
17404af
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 24, 2025
130046f
doc string for further testcase
chandra-siri Sep 24, 2025
0c61e87
remove unwanted comments
chandra-siri Sep 24, 2025
027758c
remove testing code
chandra-siri Sep 24, 2025
570983f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 24, 2025
99ffe2d
fix doc strings
chandra-siri Sep 24, 2025
e571dd1
Merge branch 'bidi_reads_6_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 24, 2025
962b7b9
instantiate `socket_like_rpc` while opening
chandra-siri Sep 24, 2025
fdaf1ae
update doc string to describe `read_ranges` format
chandra-siri Sep 25, 2025
cf6e004
add is_stream_open and check this before "open" "close" "send" "recv"
chandra-siri Sep 25, 2025
153007d
Merge branch 'bidi_reads_6_read_obj_stream' of github.com:googleapis/…
chandra-siri Sep 25, 2025
f1874ed
assert stream is indeed closed/open on close/open
chandra-siri Sep 25, 2025
ce7a73b
feat(zb-experimental): Implement `close` functionality for MRD
chandra-siri Sep 25, 2025
2383d63
feat(samples): Add multi_range_downloader samples
chandra-siri Sep 25, 2025
77e7167
Add samples for multi_range_downloader
chandra-siri Sep 25, 2025
3739091
Revert "feat(samples): Add multi_range_downloader samples"
chandra-siri Sep 25, 2025
1da749a
Merge branch 'main' of github.com:googleapis/python-storage into bidi…
chandra-siri Sep 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -71,23 +71,16 @@ class AsyncMultiRangeDownloader:
client, bucket_name="chandrasiri-rs", object_name="test_open9"
)
my_buff1 = open('my_fav_file.txt', 'wb')
my_buff1 = open('my_fav_file.txt', 'wb')
my_buff2 = BytesIO()
my_buff3 = BytesIO()
my_buff4 = any_object_which_provides_BytesIO_like_interface()
results_arr, error_obj = await mrd.download_ranges(
my_buff4 = any_object_which_provides_BytesIO_like_interface()
results_arr, error_obj = await mrd.download_ranges(
[
# (start_byte, bytes_to_read, writeable_buffer)
# (start_byte, bytes_to_read, writeable_buffer)
(0, 100, my_buff1),
(100, 20, my_buff2),
(200, 123, my_buff3),
(300, 789, my_buff4),
(100, 20, my_buff2),
(200, 123, my_buff3),
(300, 789, my_buff4),
]
)
if error_obj:
Expand All @@ -101,17 +94,6 @@ class AsyncMultiRangeDownloader:
for result in results_arr:
print("downloaded bytes", result)

if error_obj:
print("Error occurred: ")
print(error_obj)
print(
"please issue call to `download_ranges` with updated"
"`read_ranges` based on diff of (bytes_requested - bytes_written)"
)

for result in results_arr:
print("downloaded bytes", result)


"""

Expand Down
53 changes: 53 additions & 0 deletions samples/snippets/experimental/storage_download_ranges_async.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import (
AsyncMultiRangeDownloader,
)
from google.cloud.storage._experimental.asyncio.async_grpc_client import (
AsyncGrpcClient,
)
from io import BytesIO
import asyncio
import argparse


async def test_mrd(bucket_name, object_name, generation_number=None):
client = AsyncGrpcClient()._grpc_client

mrd = AsyncMultiRangeDownloader(client, bucket_name, object_name, generation_number)
await mrd.open()
# create buffers.
# Make sure buffer desitnation has enough space to accomodate bytes requetsed.
# Buffers could be in-memory or on disk
my_buff1 = open(f"sample_file_to_write_contents.txt", "wb")
my_buff2 = BytesIO()
my_buff3 = BytesIO()
my_buff4 = BytesIO()
results_arr, error_obj = await mrd.download_ranges(
[
(0, 100, my_buff1),
(100, 20, my_buff2),
(200, 123, my_buff3),
(300, 789, my_buff4),
]
)
if error_obj:
print("Error occurred: ")
print(error_obj)
print(
"please issue call to `download_ranges` with updated"
"`read_ranges` based on diff of (bytes_requested - bytes_written)"
)

for result in results_arr:
print("downloaded bytes", result)

# close MRD
await mrd.close()


if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--bucket_name", type=str, required=True)
parser.add_argument("--object_name", type=str, required=True)
parser.add_argument("--generation_number", type=int, default=None)
args = parser.parse_args()
asyncio.run(test_mrd(args.bucket_name, args.object_name, args.generation_number))