Skip to content

Modifications to JDP proxy to allow necessary KBERDL transfers#216

Open
jeff-cohere wants to merge 14 commits into
mainfrom
add-jdp-datasets-search-param
Open

Modifications to JDP proxy to allow necessary KBERDL transfers#216
jeff-cohere wants to merge 14 commits into
mainfrom
add-jdp-datasets-search-param

Conversation

@jeff-cohere
Copy link
Copy Markdown
Collaborator

@jeff-cohere jeff-cohere commented May 14, 2026

This PR adds some features that allow us to transfer files to KBERDL:

  • a datasets JDP search parameter is added to the set of JDP-specific parameters. We needed this to get the file IDs for an IMG Virus dataset.
  • Batch requests for file metadata (via the unofficial search/by_file_ids endpoint) seem flaky, sometimes dropping handfuls of metadata in the response, so a more elaborate request structure gathers missing ones (not perfect, but seems good enough for now).
  • Requests to unarchive data from tape are now batched as well, in sets of 1000 files.

Closes #220

@jeff-cohere jeff-cohere requested a review from mattldawson May 14, 2026 21:07
@jeff-cohere jeff-cohere added the JDP Involves a feature request or bugfix to the JGI Data Portal label May 14, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 14, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://kbase.github.io/dts/pr-preview/pr-216/

Built to branch gh-pages at 2026-05-26 21:00 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@jeff-cohere jeff-cohere changed the title Add datasets to the set of JDP-specific search parameters Modifications to JDP proxy to allow necessary KBERDL transfers May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

JDP Involves a feature request or bugfix to the JGI Data Portal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DTS staging requests for JDP return 404 for some large transfers

2 participants