Adding large payload support for the standalone SDK#280
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds “large payload” support to the standalone Durable Task Java SDK by introducing an Azure Blob Storage–backed payload externalization module and wiring gRPC interceptors + orchestrator-response chunking into the core client/worker to avoid gRPC message size limits.
Changes:
- Introduces new
:azure-blob-payloadsmodule implementing payload externalization/resolution via a gRPCClientInterceptorand an Azure Blob–backedPayloadStore. - Adds gRPC interceptor support to
DurableTaskGrpcClientBuilder/DurableTaskGrpcWorkerBuilder, plus worker capability announcement and orchestrator completion chunking. - Adds a runnable sample and unit/integration tests covering token handling, interceptor behavior, and chunking behavior.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| settings.gradle | Adds the new :azure-blob-payloads Gradle module to the build. |
| samples/src/main/java/io/durabletask/samples/LargePayloadSample.java | New sample demonstrating end-to-end large payload externalization with DTS + Azurite. |
| samples/build.gradle | Adds a runLargePayloadSample task and depends on :azure-blob-payloads. |
| internal/durabletask-protobuf/protos/orchestrator_service.proto | Updates protobuf contract (tags, rewind action, purge timeout). |
| internal/durabletask-protobuf/PROTO_SOURCE_COMMIT_HASH | Updates upstream proto source commit hash. |
| client/src/test/java/com/microsoft/durabletask/OrchestratorChunkingTest.java | Adds unit tests for worker chunking + action-size validation + capability announcement. |
| client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcWorkerBuilder.java | Adds interceptor registration, LP capability flag, and configurable chunk size. |
| client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcWorker.java | Applies interceptors, announces LP capability, and implements orchestrator-response chunking/validation. |
| client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcClientBuilder.java | Adds interceptor registration support to the client builder. |
| client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcClient.java | Applies registered interceptors to the client channel. |
| client/build.gradle | Adds grpc-inprocess for new in-process gRPC unit tests. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/PayloadStore.java | Introduces PayloadStore abstraction for out-of-band payload storage. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/PayloadStorageException.java | Defines exception type for permanent storage failures. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadWorkerExtensions.java | Adds worker-side helper methods to enable externalized payloads + capability flag. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadStorageOptions.java | Adds configuration options (threshold/max/container/auth/compression). |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadInterceptor.java | Implements interceptor that externalizes outbound payloads and resolves inbound tokens. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadClientExtensions.java | Adds client-side helper methods to enable externalized payloads. |
| azure-blob-payloads/src/main/java/com/microsoft/durabletask/azureblobpayloads/BlobPayloadStore.java | Implements Azure Blob Storage payload store with optional gzip compression. |
| azure-blob-payloads/src/test/java/com/microsoft/durabletask/azureblobpayloads/PayloadTokenTest.java | Unit tests for token encode/decode and token detection. |
| azure-blob-payloads/src/test/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadStorageOptionsTest.java | Unit tests for options defaults and validation behavior. |
| azure-blob-payloads/src/test/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadInterceptorTest.java | Unit tests for request externalization + response resolution across message types. |
| azure-blob-payloads/src/test/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadIntegrationTest.java | Integration tests requiring DTS emulator + Azurite for end-to-end validation. |
| azure-blob-payloads/src/test/java/com/microsoft/durabletask/azureblobpayloads/BlobPayloadStoreTest.java | Unit tests for blob upload/download/compression behavior using mocks. |
| azure-blob-payloads/spotbugs-exclude.xml | SpotBugs exclusions for the new module. |
| azure-blob-payloads/build.gradle | Build/test/spotbugs config and dependencies for the new module. |
client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcWorker.java
Outdated
Show resolved
Hide resolved
samples/src/main/java/io/durabletask/samples/LargePayloadSample.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadWorkerExtensions.java
Show resolved
Hide resolved
.../src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadWorkerExtensions.java
Show resolved
Hide resolved
.../src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadClientExtensions.java
Show resolved
Hide resolved
.../src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadClientExtensions.java
Show resolved
Hide resolved
...ds/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadStorageOptions.java
Show resolved
Hide resolved
...ds/src/main/java/com/microsoft/durabletask/azureblobpayloads/LargePayloadStorageOptions.java
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue describing the changes in this PR
Adding large payload support for the standalone SDK
resolves #issue_for_this_pr
Pull request checklist
CHANGELOG.md