Add interactive Qt panel to VR scene

[cpinter]

A very flexible way for showing menus in VR would be to implement off-line rendering of any Qt widget in a texture that is shown on a VTK plane actor, which can respond to virtual mouse events generated by the laser pointer (currently called probe mode in VTK) in VR.

Tasks that need to be done, start with first and only move to next one once it is done.

• Generate 2D texture with Qt of arbitrary QWidget. Done when you have a QImage or QPixmap containing a QWidget with some other widgets in it (for example a QFrame with a QVBoxLayout having a QCheckBox and a QSlider on it) so that the widget is never shown on the Windows desktop (i.e. everything happens in the background).
• Show texture generated in the previous step on a VTK plane actor (for example vtkImageActor). Done when the Qt GUI is visible in the scene shown in VR (positioning can be arbitrary at this point).
• Refactor VR interactor style according the new events design (Csaba)
• Re-add laser pointer (called probe mode in vtkOpenVRMenu*), TBD
• Send laser pointer events to Qt, re-render the widget if necessary, TBD
• Create default "pipboy" style panel with basic settings. Show up attached to left forearm if left controller is turned outside 180(ish) degrees. Make panel look like it's obviously interactive (back side not visible, slight frame on the edges, etc.)

Useful UI panels (extend as needs arise):

• "Main menu" widget that takes the user to the "apps" (below)
• Subject hierarchy (show/hide)
• Segment Editor (simplified)
• Interaction settings: group objects (moving them under the same interaction transform), toggle selectable, etc.

Related issues

Related pull requests

• ENH: Initial VR home widget implementation #100

[pieper]

💯 this will be a great feature!

[jcfr]

To follow up on this, @martinken recently added classes to VTK for adding Qt widget into the VR scene:

• vtkQWidgetRepresentation.h
• vtkQWidgetTexture.h
• vtkQWidgetWidget.h

[cpinter]

@jcfr Perfect! @mohammadrashid0917 just started working on this. We'll re-adjust the plan to use these classes. Mohammad, please hang on. In the meantime you can look at these classes.

[lassoan]

@jcfr We were lucky this time that minimal/no work was duplicated, but we need to coordinate a bit more closely to make sure we avoid parallel efforts. How could we do that? Where can we find planned VTK virtual reality developments? Do you have a category in the VTK bugtracker, labs page, …?

[jcfr]

How could we do that?

The plan described at the top of this issue is outstanding. I should probably have done a better job communicating about it with folks working on OpenVR at Kitware.

Moving forward, I think sharing the plan like the one at the top of this issue on VTK discourse would help.

Internally, the folks working on OpenVR features know that I am coordinating effort on the Slicer side and I think it worked pretty well as I was able to follow up here.

That said, I agreed that the feedback loop could be shorter.

May be having a list of contact associated with key VTK area would help ?

Where can we find planned VTK virtual reality developments?

Considering that this type of feature are driven by customer needs and often planned and executed in a short time frame, i think it is challenging for us to communicate about this features with a high level of granularity and far ahead.

[jcfr]

To follow up, here is a Slicer PR backporting the specific VTK changes to Slicer/VTK fork. See Slicer/Slicer#1141

I have checked that it compiles on Linux, after someone check that Slicer/VTK compiles on Windows and macOS, it will be integrated.