Skip to content

refactor(files): simplify FileClient with transfer adapters#137

Merged
v1r3n merged 14 commits into
mainfrom
feat/file-client-transfer-adapters
Jul 21, 2026
Merged

refactor(files): simplify FileClient with transfer adapters#137
v1r3n merged 14 commits into
mainfrom
feat/file-client-transfer-adapters

Conversation

@v1r3n

@v1r3n v1r3n commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Pull Request type

  • Bugfix
  • Feature
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • WHOSUSING.md
  • Other (please describe):

Changes in this PR

Depends on conductor-oss/conductor#1355 for the workflow-scoped File API.

Simplifies the public file API around one explicit, workflow-aware FileClient:

  • uploads return opaque conductor://file/<id> strings
  • every upload, download, and metadata operation requires a workflow ID
  • provider behavior moves into package-private S3, Azure, GCS, local, and generic HTTP transfer adapters
  • signed transfers use an isolated raw OkHttp client with redirects, cookies, auth, and interceptors disabled
  • multipart selection is automatic; S3 validates ETags and Azure uses stable block IDs
  • stream uploads are repeatable and do not close caller-owned streams
  • downloads use sibling temporary files and atomic replacement
  • retries refresh signed URLs, reconcile ambiguous completion, and preserve interruption
  • removes smart file objects and automatic TaskRunner upload/download integration
  • adds complete API, design, migration, and media-transcoder examples

Breaking change: workers now exchange raw handle strings and invoke FileClient explicitly. Workflows using {fileHandleId, fileName, contentType} objects need a coordinated worker rollout.

Issue #

Tests

  • ./gradlew :conductor-client:test --tests org.conductoross.conductor.client.FileClientTest --tests org.conductoross.conductor.client.FileTransferAdaptersTest --tests org.conductoross.conductor.client.FileClientPropertiesTest :conductor-client-spring:test --tests com.netflix.conductor.client.spring.ConductorClientAutoConfigurationTest
  • ./gradlew :conductor-client:spotlessJavaApply :conductor-client-spring:spotlessJavaApply
  • standalone media-transcoder sources compiled against the locally published SDK
  • live E2E against feat(files): enforce workflow-scoped file transfers conductor#1355: workflow completed with four raw handles and correct metadata sizes

Alternatives considered

Keeping provider backends as a public extension point would expose signed-URL protocol details and split retry/lifecycle ownership. Keeping smart file objects would continue coupling worker serialization and TaskRunner behavior to storage, so both were replaced with explicit handle strings and FileClient calls.

@v1r3n

v1r3n commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Updated the FileClient example to use one consistent annotated-worker pattern throughout:

  • all four media workers now use @WorkerTask
  • typed POJO input binding + @WorkflowInstanceIdInputParam
  • handle output mapping with @OutputParam
  • dependency-injected registration through AnnotatedWorkerExecutor.initWorkersFromInstances(...)
  • FileClient and example docs now explain the pattern step by step and link to WorkerTask.java

Verification:

  • AnnotatedWorkerTests passed
  • standalone media-transcoder sources compiled on Java 21
  • real media_transcode execution against http://localhost:8080/api completed with all four opaque file handles

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.40896% with 197 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../org/conductoross/conductor/client/FileClient.java 64.94% 73 Missing and 56 partials ⚠️
...toross/conductor/client/SignedUrlHttpTransfer.java 75.49% 20 Missing and 5 partials ⚠️
...oss/conductor/client/LocalFileTransferAdapter.java 68.00% 7 Missing and 1 partial ⚠️
...kes/conductor/client/http/OrkesAuthentication.java 75.00% 3 Missing and 4 partials ⚠️
...nductor/client/GenericHttpFileTransferAdapter.java 50.00% 5 Missing ⚠️
...va/org/conductoross/conductor/ai/AgentRuntime.java 86.95% 3 Missing ⚠️
...oss/conductor/client/AzureFileTransferAdapter.java 83.33% 3 Missing ⚠️
...oross/conductor/client/GcsFileTransferAdapter.java 70.00% 3 Missing ⚠️
...onductoross/conductor/ai/frameworks/AdkBridge.java 0.00% 2 Missing ⚠️
...ava/io/orkes/conductor/client/SchedulerClient.java 0.00% 2 Missing ⚠️
... and 9 more
Files with missing lines Coverage Δ Complexity Δ
...ss/conductor/ai/spring/AgentAutoConfiguration.java 100.00% <ø> (ø) 4.00 <0.00> (ø)
...ductoross/conductor/ai/spring/AgentProperties.java 100.00% <ø> (ø) 5.00 <0.00> (ø)
...ava/org/conductoross/conductor/ai/AgentConfig.java 90.00% <100.00%> (ø) 30.00 <0.00> (ø)
...ava/org/conductoross/conductor/ai/RunSettings.java 82.85% <100.00%> (+1.60%) 14.00 <2.00> (+2.00)
...org/conductoross/conductor/ai/enums/Framework.java 68.75% <ø> (ø) 3.00 <0.00> (ø)
...tor/ai/exceptions/CredentialNotFoundException.java 100.00% <ø> (ø) 3.00 <0.00> (ø)
...ross/conductor/ai/exceptions/WorkerStallError.java 100.00% <ø> (ø) 3.00 <0.00> (ø)
...ross/conductor/ai/execution/LocalCodeExecutor.java 58.18% <100.00%> (ø) 8.00 <0.00> (ø)
...ross/conductor/ai/frameworks/LangChain4jAgent.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
...oross/conductor/ai/frameworks/LangChainBridge.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
... and 44 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@v1r3n

v1r3n commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up cleanup: audited io.orkes.conductor.client.model.agent and removed redundant @JsonProperty annotations. Explicit mappings remain only for genuine wire-name differences (static_plan, snake_case tool fields, and is... status booleans) plus the write-only RespondBody fields that require explicit Jackson visibility. Added focused serialization/deserialization and alias tests; agent model tests, OrkesAgentClientTest, and Spotless all pass.

@v1r3n

v1r3n commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Converted all six io.orkes.conductor.client.model.agent DTOs to Lombok @Data and removed handwritten getters, setters, and toString methods. RespondBody.extraFields now uses Lombok @Getter(onMethod_ = @JsonAnyGetter), and null-to-empty response collections retain their behavior via @JsonSetter(nulls = Nulls.AS_EMPTY). JSON contract tests, OrkesAgentClientTest, and compilation pass.

@v1r3n

v1r3n commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Live validation against http://localhost:8080/api completed successfully at pushed commit dbc6fb84:

  • Agent E2E: serialized AgentRequest, deserialized StartResponse, polled AgentStatusResponse to COMPLETED, and verified LIVE_OK; temporary agent definition was deleted.
  • FileClient E2E: media_transcode completed with stream upload, annotated workers, parallel downloads/uploads, metadata reads, and four opaque conductor://file/... outputs.
  • Artifact verification: focused agent tests, Javadocs, shaded JAR, and publishToMavenLocal all pass.

Validation found and fixed one publication-only issue: Lombok onMethod_ compiled but broke Javadoc. RespondBody.extraFields now uses field-level @JsonAnyGetter with @Getter(AccessLevel.NONE), preserving flattened JSON without duplicate extraFields output.

@v1r3n
v1r3n merged commit 5e765dd into main Jul 21, 2026
11 checks passed
@v1r3n
v1r3n deleted the feat/file-client-transfer-adapters branch July 21, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants