Hover, segmentation and pan/zoom support for rr.Tensor

[dangom]

Related

• Closes Tensor view should have hover preview for values just like 2D view #4307
• Closes Zoom for tensor view #6849
• Closes [Feature Request] TensorView: Support for overlaying Segmentation Masks on 3D Tensors #12010

What

This PR introduces quality of life improvements to rr.Tensor that have been requested multiple times.

Note that I do not know much rust, so a lot of this code was crafted with help from agents. I understand that the maintainers are busy and have little bandwidth to review, but I figured I'd open this PR since the features are working, and if someone with better knowledge wants to take this on or do some testing so these features can make it into main, they might as well have a starting point.

Some things that would have to be done before merging are probably documentation and fallback cases. Currently the segmentation will not show without an AnnotationContext, which can be confusing. The hover UI could also be cleaned up a bit.

Summary

• Add segmentation-mask support for tensors with overlay opacity.
  • this is done by passing multiple tensors into a single TensorView.
• Introduce rr.Tensor hover UI in line with rr.Image (zoom preview, pixel values/labels).
• Improve tensor interaction (wheel slice scroll, pan/zoom, reset).

In the screenshot below, note the hover UI and the controllable opacity for the segmentation overlay.

Here an MWE:

import rerun as rr
import numpy as np
import rerun.blueprint as rrb

def main():
    rr.init("segmentation_overlay_test", spawn=True)

    # 1. Define the Annotation Context
    # This defines how the integer labels in the segmentation tensor are colored.
    rr.log("/", rr.AnnotationContext([
        rr.AnnotationInfo(id=0, label="Background", color=(0, 0, 0)),
        rr.AnnotationInfo(id=1, label="Brain Tissue", color=(255, 100, 100)), # Opaque red
        rr.AnnotationInfo(id=2, label="Ventricles", color=(100, 100, 255)),  # Opaque blue
    ]), static=True)

    # 2. Create synthetic 3D data (Z, Y, X)
    shape = (32, 128, 128)
    z, y, x = np.ogrid[:shape[0], :shape[1], :shape[2]]

    # Base MRI: A fuzzy sphere
    dist_sq = (x - 64)**2 + (y - 64)**2 + (z - 16)**2
    mri = np.exp(-dist_sq / (2 * 40**2)).astype(np.float32)
    mri += np.random.normal(0, 0.05, shape) # Add noise

    # Segmentation: Integer labels
    seg = np.zeros(shape, dtype=np.uint16)
    seg[dist_sq < 45**2] = 1 # Brain
    seg[dist_sq < 15**2] = 2 # Ventricles

    # 3. Log the tensors
    # We name the dimensions so the Tensor View can map them correctly
    dim_names = ["z", "y", "x"]
    rr.log("volume/mri", rr.Tensor(mri, dim_names=dim_names))
    rr.log("volume/segmentation", rr.Tensor(seg, dim_names=dim_names, opacity=0.5))

    # 4. Set up a Blueprint to show both in the same Tensor View
    # My changes allow the TensorView to consume multiple entities.
    blueprint = rrb.Blueprint(
        rrb.TensorView(
            name="Slicer Overlay",
            origin="volume",
            contents=["volume/mri", "volume/segmentation"]
        )
    )
    rr.send_blueprint(blueprint)

if __name__ == "__main__":
    main()

[github-actions]

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.

[Wumpf]

@dangom thanks for sharing this and kindly providing the context how you got there! You're right we're a bit tight on bandwidth now, the Tensor view in particular hasn't been a priority. I'm also unsure how we want to design mixing in annotation masks in the tensor view 🤔
But very nice to share your solution and happy it works for you out so far, at the very least we can use this to check on desired behavior when getting to it :)