Skip to content

feat(synology): implement chunked upload for large videos#1488

Merged
mengyushen merged 3 commits into
dtinit:masterfrom
SynologyOpenSource:synology-chunk-upload
Apr 2, 2026
Merged

feat(synology): implement chunked upload for large videos#1488
mengyushen merged 3 commits into
dtinit:masterfrom
SynologyOpenSource:synology-chunk-upload

Conversation

@simonxander
Copy link
Copy Markdown
Collaborator

@simonxander simonxander commented Apr 2, 2026

Goal

The goal of this change is to provide a more robust upload mechanism for large video files in SynologyDTPService. By switching from a single-stream upload to a chunked upload process, we improve reliability for very large transfers and align with the Synology C2 API's preferred method for handling significant media payloads.

Changes

  • Chunked Upload Logic: Refactored createVideo to use a multi-step upload process:
    • uploadVideoChunks: Reads the video stream in 50MB increments and uploads each chunk sequentially to the new /import/item/chunk endpoint.
    • completeVideoUpload: Sends a final request to /import/item/complete with the total chunk count and metadata (title, description, timestamp) to finalize the file.
  • API Configuration:
    • Updated C2Api and synology.yaml to include paths for the new chunk and completion endpoints.
  • Client Optimization:
    • Updated configureClient to force HTTP/1.1 and increased the default read timeout to 120 seconds to ensure stable long-running connections during chunk transmission.
    • Simplified sendPostRequest by removing the manual timeout override, relying instead on the pre-configured client.
  • Memory Efficiency: Reuses a single byte array buffer for chunking to minimize heap allocations during the transfer of large files.
  • Others: Move file content to the end of multipart

Testing

  • OOM Validation: Updated SynologyDTPServiceOOMTest to verify that a 1GB video is correctly split into multiple chunks and uploaded without exceeding memory limits.
  • Functional Tests: Updated SynologyDTPServiceTest to accommodate the new two-step upload flow (Multipart chunks followed by a FormBody completion) and added a specific case shouldSendMultipleChunksForLargeVideo to verify correct indexing.

## Goal
The goal of this change is to provide a more robust upload mechanism for large video files in `SynologyDTPService`. By switching from a single-stream upload to a chunked upload process, we improve reliability for very large transfers and align with the Synology C2 API's preferred method for handling significant media payloads.

## Changes
- **Chunked Upload Logic:** Refactored `createVideo` to use a multi-step upload process:
    - `uploadVideoChunks`: Reads the video stream in 50MB increments and uploads each chunk sequentially to the new `/import/item/chunk` endpoint.
    - `completeVideoUpload`: Sends a final request to `/import/item/complete` with the total chunk count and metadata (title, description, timestamp) to finalize the file.
- **API Configuration:**
    - Updated `C2Api` and `synology.yaml` to include paths for the new chunk and completion endpoints.
- **Client Optimization:**
    - Updated `configureClient` to force **HTTP/1.1** and increased the default read timeout to 120 seconds to ensure stable long-running connections during chunk transmission.
    - Simplified `sendPostRequest` by removing the manual timeout override, relying instead on the pre-configured client.
- **Memory Efficiency:** Reuses a single byte array buffer for chunking to minimize heap allocations during the transfer of large files.

## Testing
- **OOM Validation:** Updated `SynologyDTPServiceOOMTest` to verify that a 1GB video is correctly split into multiple chunks and uploaded without exceeding memory limits.
- **Functional Tests:** Updated `SynologyDTPServiceTest` to accommodate the new two-step upload flow (Multipart chunks followed by a FormBody completion) and added a specific case `shouldSendMultipleChunksForLargeVideo` to verify correct indexing.
@simonxander simonxander force-pushed the synology-chunk-upload branch from 04fdd06 to 4faa5de Compare April 2, 2026 11:08
@mengyushen mengyushen merged commit 0ad82b1 into dtinit:master Apr 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants