feat: skip completed files in resumable dataset uploads by carloea2 · Pull Request #5929 · apache/texera

carloea2 · 2026-06-24T07:37:14Z

What changes were proposed in this PR?

This PR improves dataset upload retry behavior by distinguishing between two cases that currently look similar to users:

Case	Behavior
Active multipart upload session exists	Ask whether to resume or restart the upload.
Matching file already exists in the dataset	Ask whether to upload again or skip it.

The backend adds a dataset endpoint that checks candidate upload files against committed and staged dataset files by path and size. The frontend uses that endpoint after checking active multipart sessions, so failed uploads can still be resumed while completed matching files can be skipped from the retry batch.

The user-facing copy intentionally says a file with the same path and size exists, instead of implying byte-for-byte certainty.

Any related issues, documentation, discussions?

Related to discussion #5744: Improve resumable upload: track completion at the batch/session level.

How was this PR tested?

Added backend and frontend tests covering:

Test	Coverage
`DatasetResourceSpec`	Finds committed and staged files only when path and size match.
`DatasetService` spec	Forces one multipart part upload to fail, then verifies retry uploads only the missing part.
`FilesUploaderComponent` spec	Verifies a mixed retry batch asks Resume/Restart for a failed multipart file and Upload/Skip for a completed matching file.

Commands run:

sbt "FileService/testOnly org.apache.texera.service.resource.DatasetResourceSpec -- -z findExistingUploadFiles"

yarn ng run gui:test --include=src/app/dashboard/service/user/dataset/dataset.service.spec.ts

yarn ng run gui:test --include=src/app/dashboard/component/user/files-uploader/files-uploader.component.spec.ts

yarn eslint src/app/dashboard/component/user/files-uploader/files-uploader.component.ts src/app/dashboard/component/user/files-uploader/files-uploader.component.spec.ts src/app/dashboard/component/user/files-uploader/conflicting-file-modal-content/conflicting-file-modal-content.component.ts src/app/dashboard/service/user/dataset/dataset.service.ts src/app/dashboard/service/user/dataset/dataset.service.spec.ts

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Me

github-actions · 2026-06-24T07:37:34Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @Yicong-Huang, @xuang7, @Ma77Ball
You can notify them by mentioning @Yicong-Huang, @xuang7, @Ma77Ball in a comment.

codecov-commenter · 2026-06-24T07:40:08Z

Codecov Report

❌ Patch coverage is 79.16667% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.73%. Comparing base (434bcae) to head (dd12294).

Files with missing lines	Patch %	Lines
...nt/user/files-uploader/files-uploader.component.ts	81.57%	2 Missing and 5 partials ⚠️
...ache/texera/service/resource/DatasetResource.scala	57.14%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5929      +/-   ##
============================================
- Coverage     56.93%   56.73%   -0.21%     
+ Complexity     3026     3005      -21     
============================================
  Files          1129     1129              
  Lines         43794    43854      +60     
  Branches       4743     4771      +28     
============================================
- Hits          24936    24880      -56     
- Misses        17384    17488     +104     
- Partials       1474     1486      +12

Flag	Coverage Δ		*Carryforward flag
access-control-service	`70.00% <ø> (ø)`
agent-service	`44.59% <ø> (ø)`		Carriedforward from 875ef96
amber	`57.78% <ø> (-0.99%)`	⬇️	Carriedforward from 875ef96
computing-unit-managing-service	`0.00% <ø> (ø)`
config-service	`52.30% <ø> (ø)`
file-service	`63.97% <57.14%> (+1.16%)`	⬆️
frontend	`50.57% <82.92%> (+0.44%)`	⬆️
notebook-migration-service	`78.57% <ø> (ø)`
pyamber	`90.20% <ø> (ø)`		Carriedforward from 875ef96
python	`90.76% <ø> (ø)`		Carriedforward from 875ef96
workflow-compiling-service	`55.14% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

xuang7 · 2026-06-24T18:55:51Z

Hi @carloea2, thanks for the PR. Could you open a corresponding issue for this PR? The existing discussion may not include the implementation design details for this PR.

Also, I was wondering whether the notification window might overlap with the resumable/restart flow for incomplete uploads.

Thanks!

carloea2 · 2026-06-24T19:55:07Z

What do you mean with overlap?

Issue: #5938

Copilot

Pull request overview

This PR improves resumable dataset uploads by distinguishing “resumable multipart session exists” from “file already exists in the dataset (same path + size)”, enabling users to skip already-completed files during a retry batch.

Changes:

Backend: add POST /dataset/{did}/existing-upload-files to detect already-present committed/staged files by normalized path + size.
Frontend: extend the upload retry flow to first handle active multipart sessions, then prompt Upload/Skip for completed matching files.
Tests: add/extend Scala + Angular unit tests to cover the new endpoint and the mixed retry-batch UX.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
frontend/src/app/dashboard/service/user/dataset/dataset.service.ts	Adds client method to query existing upload files by path + size.
frontend/src/app/dashboard/service/user/dataset/dataset.service.spec.ts	Adds coverage for existing-upload-files call and multipart resume behavior.
frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.html	Wires dataset id (`did`) into the uploader component.
frontend/src/app/dashboard/component/user/files-uploader/files-uploader.component.ts	Adds Upload/Skip prompt for already-existing files and integrates both checks in retry flow.
frontend/src/app/dashboard/component/user/files-uploader/files-uploader.component.spec.ts	Adds new tests for mixed retry batches and Upload/Skip For All behavior.
frontend/src/app/dashboard/component/user/files-uploader/conflicting-file-modal-content/conflicting-file-modal-content.component.ts	Extends modal data with optional `hint`.
frontend/src/app/dashboard/component/user/files-uploader/conflicting-file-modal-content/conflicting-file-modal-content.component.html	Renders a custom hint when provided.
file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala	Adds backend tests for matching by path+size and validation/auth checks.
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala	Implements the new existing-upload-files endpoint.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

carloea2 · 2026-06-26T06:59:43Z

+      val requested = Option(request)
+        .map(_.files)
+        .getOrElse(List.empty)
+        .map { file =>


carloea2 · 2026-06-26T06:59:54Z

+      val committed = getLatestDatasetVersion(ctx, did)
+        .map(v =>
+          LakeFSStorageClient
+            .retrieveObjectsOfVersion(dataset.getRepositoryName, v.getVersionHash)
+            .map(obj => obj.getPath -> obj.getSizeBytes.longValue())
+        )
+        .getOrElse(List.empty)


carloea2 · 2026-06-26T07:00:02Z

+      val staged = LakeFSStorageClient
+        .retrieveUncommittedObjects(dataset.getRepositoryName)
+        .filterNot(diff => Option(diff.getType).exists(_.getValue.equalsIgnoreCase("removed")))
+        .flatMap(diff => Option(diff.getSizeBytes).map(size => diff.getPath -> size.longValue()))


xuang7

Overall LGTM!

One small question: if a file appears in both activePaths and existingPaths, would the user first see the Resume/Restart dialog and then the Upload/Skip dialog for the same file? Having two dialogs back to back could be confusing. Would it be better to exclude active uploads from the existingPaths conflict flow?

xuang7 · 2026-06-29T17:49:33Z

+
+      val existing = (committed ++ staged).toMap
+      val matches = requested
+        .collect { case (path, size) if existing.get(path).contains(size) => path }


The backend matches on the normalized path and returns that normalized key, but the frontend compares against the original item.name. Suggest either echoing back the original request path from the backend (use the original path as the returned value), or normalizing symmetrically on the frontend.

xuang7 · 2026-06-29T17:50:51Z

  }

+  private async resolveExistingFiles(items: FileUploadItem[], existingPaths: string[]): Promise<FileUploadItem[]> {
+    const existing = new Set(existingPaths ?? []);


frontend compares against the original item.name.

carloea2 · 2026-06-30T22:58:27Z

Thanks @xuang7. I added tests for both cases.

For the overlap case, I kept the current behavior intentionally: if the same file appears in both activePaths and existingPaths, the user gets Resume/Restart first and then Upload/Skip, because those are two separate decisions. The new frontend spec covers that exact sequence.

For the path mismatch, I updated the backend to normalize for lookup but return the original requested path in filePaths, so the frontend comparison against item.name remains stable. I added a backend test for that too.

github-actions Bot assigned carloea2 Jun 24, 2026

github-actions Bot added frontend Changes related to the frontend GUI platform Non-amber Scala service paths labels Jun 24, 2026

carloea2 force-pushed the batch-resumable-upload branch from 94ce43f to ce97bb4 Compare June 24, 2026 07:48

feat: skip completed dataset uploads

a586457

carloea2 force-pushed the batch-resumable-upload branch from ce97bb4 to a586457 Compare June 24, 2026 18:41

Merge branch 'main' into batch-resumable-upload

fd99789

carloea2 mentioned this pull request Jun 24, 2026

Improve dataset upload retries for completed and incomplete files #5938

Open

6 tasks

aglinxinyuan requested a review from Copilot June 25, 2026 22:49

Copilot started reviewing on behalf of aglinxinyuan June 25, 2026 22:49 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

carloea2 added 3 commits June 26, 2026 00:57

fix(file-service): harden existing upload checks

9815b96

Merge branch 'main' into batch-resumable-upload

53bfa9c

Merge branch 'main' into batch-resumable-upload

97e0fef

xuang7 approved these changes Jun 29, 2026

View reviewed changes

fix(file-service): preserve upload request paths

4d90cd5

carloea2 added 2 commits June 30, 2026 15:58

Merge branch 'main' into batch-resumable-upload

4346db4

Merge branch 'main' into batch-resumable-upload

dd12294

Uh oh!

Conversation

carloea2 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Uh oh!

github-actions Bot commented Jun 24, 2026

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xuang7 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carloea2 commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

carloea2 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

carloea2 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

carloea2 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

xuang7 left a comment

Choose a reason for hiding this comment

Uh oh!

xuang7 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

carloea2 Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

xuang7 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

carloea2 Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

carloea2 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

carloea2 commented Jun 24, 2026 •

edited

Loading

codecov-commenter commented Jun 24, 2026 •

edited

Loading

xuang7 commented Jun 24, 2026 •

edited

Loading