Automatically use zip storage if the .ozx output file extension is used#322
Draft
melissalinkert wants to merge 5 commits into
Draft
Automatically use zip storage if the .ozx output file extension is used#322melissalinkert wants to merge 5 commits into
.ozx output file extension is used#322melissalinkert wants to merge 5 commits into
Conversation
Member
Author
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #164, see https://ngff.openmicroscopy.org/rfc/9/index.html. Not super high priority to review, just had a branch in progress that I didn't want to forget about.
This required updating to zarr-java 0.1.3 due to zarr-developers/zarr-java#73. A side effect of that was needing to replace all usage of
byteshufflewithshuffle, but that seems to be expected based on https://github.com/scalableminds/blosc-java/compare/ca183797fcb7df2ae61323141108235a41f62076..ubuntu-arm64#diff-7b6ef77c00daf6ad090dbc1d01878b090b49803117075a1b0b24f1dbde000bbeR259 (cc @normanrz).Conversion behavior should be unchanged with default NGFF version (
0.4, i.e. Zarr v2). If--ngff-version 0.5is used and the output file name ends with.ozx, then the resulting output should be a single file (not a directory) that contains the whole dataset. raw2ometiff does not yet support the.ozxfile as input, but unzipping the.ozxfile to its own.zarrdirectory and running raw2ometiff on that directory should work as normal. Using--ngff-version 0.5without the.ozxextension should also continue to work as normal.I did need to change a bit how the
OME/METADATA.ome.xmlfile is written, since everything now needs to be written through theStoreinstance or it won't end up in the.ozxfile. When trying the non-zip output, it would be good to double-check that theOME/METADATA.ome.xmlfile still looks fine.I think we could technically use this feature with NGFF 0.4/Zarr v2, but RFC-9 doesn't apply to that version of the spec, so maybe that's making things too complicated.
I did not extensively test this with larger datasets, so would be good to include some larger slides and plates when doing functional testing, in particular things that might cause the output file to exceed 4GB.