Support OME-zarr labels layers

[will-moore]

It would be great to support ome-zarr labels data.
This is a zarr image where values of 0 are transparent (no label) and colour of each label may be specified by metadata:
https://github.com/ome/omero-ms-zarr/blob/master/spec.md#image-label-metadata

I can imagine that loading the zarr data into new layers won't need much new code (happy to have a stab at this), but I don't know how to render particular pixel values with pre-defined colours?
It's important that the labels layer is transparent so that the underlying image is visible, so I believe they would have to be part of the same Deck.gl layer?

See ome-zarr labels support in napari, added via ome-zarr-py plugin: ome/ome-zarr-py#30

These labels layers may be part of the same ome-zarr that contains image layers (see spec) but it's possible that you might load image layers from one source and labels from another.

Having just been reading #32, I can imagine that an API for showing/hiding or setting colors on masks would be very cool. Similar to what you're doing in vitessce http://vitessce.io/?dataset=linnarsson-2018

cc @joshmoore

[will-moore]

Another example of showing / hiding / coloring labels: https://downloads.openmicroscopy.org/presentations/2020/community-call-2020-10-29/OMERO%20Plus,%20PathViewer%20and%20NGFF%202020-10-27.mp4

[manzt]

Ah, I wasn't aware that the labels data were dense arrays. What are the non-zero elements for the labels (value wise)? I think we should be able to reuse the loader modules (to fetch data tiles), but I think transparaent for 0 values will require a new layer (or shaders at minimum) perhaps @ilan-gold has some input.

[manzt]

Having just been reading #32, I can imagine that an API for showing/hiding or setting colors on masks would be very cool. Similar to what you're doing in vitessce http://vitessce.io/?dataset=linnarsson-2018

I think we use a deck.gl polygon layer for the segmentation in vitessce because the data are in a columnar format (not dense pixel arrays). Both @keller-mark and @ilan-gold have more experience with writing custom layers in deck.gl, and hopefully they have some ideas of how to do something similar for dense array masks

With that said, I like the idea of having different layers because then we can render UI elements based on different layerProps for each layer.

[ilan-gold]

Yes we use selectable polygon layers from deck.gl which requires pre-processing the data to get polygons but likely will not require any sort of pyramid/multi-resolution handling.

We could certainly write shaders to handle this concept of value-per-color, raster/pixel-based-storage, segmentation mask, especially if this is the direction OMERO is headed - it would be very straightforward to do and could scale very well to lots of different labels if what I have in my head is right for the implementation. I agree @manzt that we should be able to reuse the loaders for it - we would change the layer (or really just the shaders) used by XRLayer to accommodate this new data and reuse the MultiscaleImageLayer with a few tweaks to the APIs.

[manzt]

We could certainly write shaders to handle this concept of value-per-color, raster/pixel-based-storage, segmentation mask, especially if this is the direction OMERO is headed

See Will's link above (https://github.com/ome/omero-ms-zarr/blob/master/spec.md#image-label-metadata). The OME-Zarr specification uses (dense) zarr arrays for masks.

[manzt]

@will-moore

Ilan and I talked about this today and I think we have an idea for implementation (likely a new layer with custom shaders). I would be useful to have an example dataset with these arrays to work with. I can't seem to find the an image with "image-label" metadata? Is this different from "masks"?

I've tried opening the examples in ome/ome-zarr-py#30 with the ome-zarr-py plugin and the masks don't seem to load the masks layers. Again, I'm not sure if mask / image-label are different.

$ napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr # just opens image layers

[will-moore]

Yeah, I had to ask around to find an image with labels. (we are planning a directory page to list all our ome-zarr images soon).
This one has labels:

napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr

The presence of labels is specified by the existence of https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr/labels/.zattrs

Labels are multi-resolution arrays (same as pixel data) but tiles may be missing if they are all zeros to save space when you have sparsely-populated labels.

[jluethi]

Hey @manzt
Super cool project and great to see the Vizarr work for Zarr V3!
I was wondering if this Label layer support for Vizarr had continued at all after 2020 or whether today's state is still at "it would need a new layer with custom shaders".
Is there interest in this? Is this anyhow on a roadmap? Would there be interest in contributions for this?

[manzt]

Label layer support still requires shader changes. With Viv migrating to deck.gl 9, contributing directly to Vizarr might not be ideal due to potential breaking changes.

I'm focused on wrapping up my PhD and may not have the time to push this feature forward, but I'm happy to support the work. That said, I'd say it's better to focus on a feature like this after we've been able to upgrade Viv/Deck.gl.

[jluethi]

Hey @manzt
Thanks for the feedback! And fingers crossed for wrapping up the PhD!

The major question for me is: How large of an undertaking do you expect adding such label layer shader changes would be? Any pointers & support from your side on how to best go about it would be appreciated. And thanks also for the context on the upcoming migration. Is there a potential timeline for the Viv/deck.gl change? Is that a major tricky migration or something coming in the next few weeks to months?

I'd be happy to coordinate with you and @zonia3000 from our team on how we could contribute to the label layer shader work if that makes sense. Sonia has already been looking into the vizarr code a little bit in the context of enabling us to use the vizarr viewer in the Fractal platform (see https://github.com/fractal-analytics-platform/fractal-vizarr-viewer). It has been impressive to see what vizarr can do out-of-the-box! 👏🏻

[ilan-gold]

Is there a potential timeline for the Viv/deck.gl change? Is that a major tricky migration or something coming in the next few weeks to months?

It will happen within the next two months or so, I imagine, but it's tough to say. We discovered bugs in their codebase and may continue to do so. So this means we can't just do it so quickly.

How large of an undertaking do you expect adding such label layer shader changes would be? Any pointers & support from your side on how to best go about it would be appreciated.

It should not be too hard; for an example, see vitessce

[jluethi]

Thanks @ilan-gold that's very useful to hear. Fingers crossed for the deck.gl change then! 🤞🏻

[tcompa]

Label layer support still requires shader changes. With Viv migrating to deck.gl 9, contributing directly to Vizarr might not be ideal due to potential breaking changes.

I'm focused on wrapping up my PhD and may not have the time to push this feature forward, but I'm happy to support the work. That said, I'd say it's better to focus on a feature like this after we've been able to upgrade Viv/Deck.gl.

Hi there, if we understand correctly the deck.gl 9 update for viv has progressed. Would it be a good time for resuming the discussion about labels (cc @manzt)? We at the Fractal team (cc @zonia3000 @jluethi) would be interested in exploring this issue.

We understand the basic structure of OME-Zarr images, and we have example data with labels (on top of the many public examples), but we would likely need some advice/support re: integration in vizarr. In the current discussion, the most specific plan seems to be #45 (comment) by @ilan-gold - is this something relevant at this point? It'd be great if we could get some more details.

To get started, we would try to implement the core functionality only - namely we would postpone:

1. Discovery of available label images (which eventually should be based on the metadata in the labels group).
2. Interactive selection and control of labels (e.g. picking which label images should be displayed from a dropdown menu), which depends on the labels discovery.
3. Any implementation as part of the plate/well loaders - which we will evaluate at a later point, since these parts seem to be handled differently in vizarr.

Any feedback is welcome - thank you!

[manzt]

Thanks for keeping the conversation going and for opening this thread! My PhD defense is scheduled for next week, so I won’t be able to contribute actively until after that.

I appreciate your understanding and look forward to diving into this discussion once I’m available.

[tcompa]

No worries, and all the best for your PhD defense!
We'll get in touch again a bit later.