Experimental: AWS S3 storage driver by jmaeagle99 · Pull Request #1388 · temporalio/sdk-python

jmaeagle99 · 2026-03-23T22:30:50Z

What was changed

Add an AWS S3 storage driver integration module that ships with the Python SDK.
- Driver can consume arbitrary clients via an S3StorageDriverClient implementation.
- Module provide a client implementation for the aioboto3 library.
- Driver allows static setting of the bucket or a callable to dynamically determine the name of the target bucket for each uploaded payload.
- Driver uploads payloads using content addressable storage under a structured object key format that allows for external lifecycle management.
- Driver checks for key existence before attempting to upload large payloads.
- Driver verifies integrity of externally retrieved payload as modified content would impact workflow determinism.
Add an aws-s3 extra to the temporalio package to assist in including the dependencies for the aioboto3 client implementation.
Add unit and integration tests that use the moto server to simulate an S3 server.

Why?

Provide a default driver for externally storing large payloads in AWS S3.

Checklist

Closes S3 External Storage Driver - Python #1390
How was this tested:

Existing tests
New S3 driver unit test and integration tests via moto AWS S3 HTTP service emulator
Manual testing targeting real AWS S3 bucket

Any docs updates needed? Yes

jmaeagle99 · 2026-03-25T17:34:19Z

temporalio/contrib/aws/s3driver/__init__.py

@@ -0,0 +1,13 @@
+"""Amazon S3 storage driver for Temporal external payload storage."""


Add experimental

jmaeagle99 · 2026-03-25T17:37:49Z

temporalio/contrib/aws/s3driver/_driver.py

+        self,
+        client: S3StorageDriverClient,
+        bucket: str | Callable[[StorageDriverStoreContext, Payload], str],
+        driver_name: str | None = None,


Set default here and validate non-empty in init

jmaeagle99 · 2026-03-25T17:39:50Z

temporalio/contrib/aws/s3driver/_driver.py

+                a bucket name. A callable allows dynamic per-payload bucket
+                selection.
+            driver_name: Name of this driver instance. Defaults to
+                ``"aws.s3driver"``. Override only when registering


Only need to override when registering multiple of the same type.

jmaeagle99 · 2026-03-25T17:59:00Z

temporalio/contrib/aws/s3driver/_driver.py

+
+            digest_segments = f"/d/sha256/{hash_digest}"
+
+            key = f"v0{namespace_segments}{context_segments}{digest_segments}"


Validate that missing segments or invalid characters are sanitized.

Especially for user provided information such as workflow ID and activity ID.

jmaeagle99 · 2026-03-25T18:11:03Z

temporalio/contrib/aws/s3driver/_driver.py

+           This API is experimental.
+    """
+
+    def __init__(


Extra config knobs:

Disable hash check

Disable object existence check

jmaeagle99 · 2026-03-25T18:19:28Z

temporalio/contrib/aws/s3driver/_client.py

+def new_aioboto3_client(client: S3Client) -> Aioboto3StorageDriverClient:
+    """Create an :class:`S3StorageDriverClient` from an aioboto3 S3 client.
+
+    This is a convenience factory. Equivalent to ``Aioboto3StorageDriverClient(client)``.


jmaeagle99 · 2026-03-25T18:20:25Z

temporalio/contrib/aws/s3driver/_client.py

+    async def object_exists(self, *, bucket: str, key: str) -> bool:
+        """Check existence via aioboto3's ``head_object``."""
+        try:
+            await self._client.head_object(Bucket=bucket, Key=key)


Is there an exists method that doesn't raise?

jmaeagle99 · 2026-03-25T18:21:01Z

temporalio/contrib/aws/s3driver/README.md

+async with session.client("s3") as s3_client:
+    driver = S3StorageDriver(client=s3_client, bucket="my-temporal-payloads")


Fix to use new_aioboto3_client method

jmaeagle99 · 2026-03-25T18:21:56Z

README.md

-    ),
-)
+session = aioboto3.Session()
+async with session.client("s3") as s3_client:


use new_aioboto3_client method

jmaeagle99 · 2026-03-25T18:24:00Z

tests/worker/test_s3driver.py

@@ -0,0 +1,454 @@
+"""Worker integration tests for S3StorageDriver key structure.


Move to tests/contrib/aws/s3driver

jmaeagle99 force-pushed the s3-driver-extra branch 5 times, most recently from 658d2e1 to 5d6cd7e Compare March 25, 2026 05:37

jmaeagle99 marked this pull request as ready for review March 25, 2026 05:43

jmaeagle99 requested a review from a team as a code owner March 25, 2026 05:43

jmaeagle99 changed the title ~~Experimental: AWS S3 driver extra package~~ Experimental: AWS S3 storage driver Mar 25, 2026

Experimental: AWS S3 driver extra package

66a3fea

jmaeagle99 force-pushed the s3-driver-extra branch from 5d6cd7e to 66a3fea Compare March 25, 2026 05:59

jmaeagle99 commented Mar 25, 2026

View reviewed changes

		@@ -0,0 +1,13 @@
		"""Amazon S3 storage driver for Temporal external payload storage."""


		digest_segments = f"/d/sha256/{hash_digest}"

		key = f"v0{namespace_segments}{context_segments}{digest_segments}"

		async with session.client("s3") as s3_client:
		driver = S3StorageDriver(client=s3_client, bucket="my-temporal-payloads")

		@@ -0,0 +1,454 @@
		"""Worker integration tests for S3StorageDriver key structure.

Conversation

jmaeagle99 commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jmaeagle99 commented Mar 23, 2026 •

edited

Loading