Skip to content

Modifications to JDP proxy to allow necessary KBERDL transfers#216

Merged
jeff-cohere merged 10 commits into
mainfrom
add-jdp-datasets-search-param
Jun 3, 2026
Merged

Modifications to JDP proxy to allow necessary KBERDL transfers#216
jeff-cohere merged 10 commits into
mainfrom
add-jdp-datasets-search-param

Conversation

@jeff-cohere
Copy link
Copy Markdown
Collaborator

@jeff-cohere jeff-cohere commented May 14, 2026

This PR adds some features that allow us to transfer files to KBERDL:

  • a datasets JDP search parameter is added to the set of JDP-specific parameters. We needed this to get the file IDs for an IMG Virus dataset.
  • Batch requests for file metadata (via the unofficial search/by_file_ids endpoint) seem flaky, sometimes dropping handfuls of metadata in the response, so a more elaborate request structure gathers missing ones (not perfect, but seems good enough for now).
  • Requests to unarchive data from tape are now batched as well, in sets of 1000 files.
  • A transfer ID is now assigned and returned immediately after a transfer specification has been validated, allowing pre-transfer work to be performed afterward.

Closes #219
Closes #220

@jeff-cohere jeff-cohere requested a review from mattldawson May 14, 2026 21:07
@jeff-cohere jeff-cohere added the JDP Involves a feature request or bugfix to the JGI Data Portal label May 14, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 14, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://kbase.github.io/dts/pr-preview/pr-216/

Built to branch gh-pages at 2026-06-03 18:11 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@jeff-cohere jeff-cohere changed the title Add datasets to the set of JDP-specific search parameters Modifications to JDP proxy to allow necessary KBERDL transfers May 26, 2026
@jeff-cohere
Copy link
Copy Markdown
Collaborator Author

I'm testing these new features in Spin, and so far it seems like everything is working properly.

@jeff-cohere jeff-cohere force-pushed the add-jdp-datasets-search-param branch from 79764c7 to 840b7eb Compare June 3, 2026 18:11
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

Codecov Report

❌ Patch coverage is 60.00000% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.37%. Comparing base (3964b63) to head (840b7eb).

Files with missing lines Patch % Lines
databases/jdp/database.go 66.00% 25 Missing and 9 partials ⚠️
transfers/dispatcher.go 26.08% 16 Missing and 1 partial ⚠️
transfers/store.go 77.77% 6 Missing ⚠️
auth/kbase_auth_server.go 0.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #216      +/-   ##
==========================================
- Coverage   59.51%   59.37%   -0.15%     
==========================================
  Files          34       34              
  Lines        4547     4605      +58     
==========================================
+ Hits         2706     2734      +28     
- Misses       1529     1553      +24     
- Partials      312      318       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jeff-cohere jeff-cohere merged commit 50fe0cf into main Jun 3, 2026
10 checks passed
@jeff-cohere jeff-cohere deleted the add-jdp-datasets-search-param branch June 3, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

JDP Involves a feature request or bugfix to the JGI Data Portal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DTS staging requests for JDP return 404 for some large transfers DTS should return a provisional transfer ID immediately after a request

2 participants