segment color shader

[chrisj]

Rather than transpiling the shader to javascript, I setup a basic shader to render onto an offscreen canvas, and using gl.readPixels to get the value back into javascript. glsl-transpiler assumes a node environment, rather than modify or use polyfills, this seemed like a simpler solution. I also have a prototype for calculating multiple segment colors with one render pass, outputting a pixel per segment, I noted in the code that get automaticLayerBarColors would be a great place to use that.

It doesn't have much purpose yet, I need to re-add my segmentaton property shader additions, but I wanted to get feedback on if this readPixels approach is preferable.

[chrisj]

I've added support for segment properties here: https://github.com/seung-lab/neuroglancer/tree/cj-color-shader-share

It is still rough, some slice view segmentation rendering options are left out for now, but it would be great to have some feedback on the format of the user shader so we could potentially start using this before the implementation is settled.

I will add an example link of h01 with no auth requirement in the following comment. Here is a short link requiring authentication that showcases coloring segments by their tags. https://cj-color-shader-share-dot-neuroglancer-dot-seung-lab.ue.r.appspot.com/#!middleauth+https://global.daf-apis.com/nglstate/api/v1/5748761140658176

I decided to try to get the color computation for all the various output(3d/2d/html) running on as much of the same code as possible. That means I used the approach necessary for slice view, having each draw call have access to all the segment properties. I have one texture that is a map from segment id to segment property index which is used to determine if we have segment property info for that segment as well as used as an index to read from the individual texture for each segment property. For now I have each tag on it's own texture for simplicity but it could be optimized later to use bit flags. Textures only exist if they are actively used by the user shader. My goal is for SegmentColorUserShaderManager to contain all of the coloring logic aside of output specific features such as highlightColor.

One difficulty is the fact that segment properties allow an id to be any string. With the h01 dataset for instance, there are property ids with a forward slash. Annotation properties are restricted to the regular expression /^[a-z][a-zA-Z0-9_]*$/. I wish that were the case for segment properties as well. I currently have a preprocessor to support prop("property_id") and tag("tag_name"). I have left out some error handling that would be necessary to completely support that feature.

The approach to use the same rendering logic between all three allows for the possibility of meshes that combine multiple segments, which could be used in the future for base segment coloring. It can also be used to efficiently calculate automaticLayerBarColors if we expand getShaderSegmentColor to take an array of segments and output to a canvas with a pixel per segment.

I should try to reuse code that generates the segment query property histograms for the invlerp control.
The tests I wrote for getShaderSegmentColor give us coverage on most of the color logic for mesh/skeleton and 2d segmentation. I would like to add tests for the rest as well, especially 2d. Another important issue to address is the UI crowding, particularly bad when you have a skeleton source so there are two editable use shaders within the render tab. Making the tab a scrollable element will help.

[chrisj]

Here is a public link that shows some more functionality of the segment color shader

https://cj-color-shader-share-dot-neuroglancer-dot-seung-lab.ue.r.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B8e-9%2C%22m%22%5D%2C%22y%22:%5B8e-9%2C%22m%22%5D%2C%22z%22:%5B3.3e-8%2C%22m%22%5D%7D%2C%22position%22:%5B358522.71875%2C86918.5%2C2660.74072265625%5D%2C%22crossSectionScale%22:115.89143046777498%2C%22projectionOrientation%22:%5B0.12615253031253815%2C0.009984897449612617%2C-0.0959339365363121%2C0.9873107671737671%5D%2C%22projectionScale%22:95374.09332403584%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22precomputed://gs://h01-release/data/20210601/4nm_raw%22%2C%22tab%22:%22rendering%22%2C%22shaderControls%22:%7B%22normalized%22:%7B%22range%22:%5B147%2C255%5D%2C%22window%22:%5B120%2C255%5D%7D%7D%2C%22name%22:%224nm%20EM%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://gs://h01-release/data/20210601/c3%22%2C%22subsources%22:%7B%22default%22:true%2C%22bounds%22:true%2C%22properties%22:true%2C%22skeletons%22:true%7D%2C%22enableDefaultSubsources%22:false%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22notSelectedAlpha%22:1%2C%22skeletonRendering%22:%7B%22lineWidth2d%22:10%7D%2C%22meshRenderScale%22:2%2C%22segmentColorShader%22:%22#uicontrol%20invlerp%20normalized%5Cnvec3%20segmentColor%28vec3%20color%2C%20bool%20hasProperties%29%20%7B%5Cn%20%20if%20%28%21hasProperties%29%20%7B%5Cn%20%20%20%20return%20vec3%280.5%2C%200.5%2C%200.5%29%3B%5Cn%20%20%7D%5Cn%20%20%5Cn%20%20if%20%28tag%28%5C%22L1%5C%22%29%29%20%7B%5Cn%20%20%20%20return%20vec3%281.0%2C%200.0%2C%200.0%29%3B%5Cn%20%20%7D%5Cn%20%20%5Cn%20%20return%20colormapCubehelix%28normalized%28%29%29%3B%5Cn%7D%5Cn%22%2C%22segments%22:%5B%2237029363761%22%2C%2228469485117%22%2C%2228003801168%22%2C%2237145660784%22%2C%2237685007129%22%5D%2C%22colorSeed%22:3871475997%2C%22shaderControls%22:%7B%22normalized%22:%7B%22range%22:%5B143%2C1030%5D%2C%22window%22:%5B0%2C1587%5D%2C%22property%22:%22NSIi%22%7D%7D%2C%22name%22:%22c3%20segmentation%22%7D%5D%2C%22showAxisLines%22:false%2C%22showSlices%22:false%2C%22selectedLayer%22:%7B%22col%22:1%2C%22size%22:444%2C%22visible%22:true%2C%22layer%22:%22c3%20segmentation%22%7D%2C%22layout%22:%22xy-3d%22%2C%22selection%22:%7B%22row%22:0%2C%22visible%22:false%7D%7D

[chrisj]

I wrote some documentation here that explains the format of the user shader:
https://github.com/seung-lab/neuroglancer/blob/cj-color-shader-share/src/layer/segmentation/rendering.md

[chrisj]

I didn't think you would have time to review the code due to the voxel painting PR but I opened up this PR here #871 so you can more easily take a look at it, it has some test deployment code

[stuarteberg]

I'm not qualified to comment on the implementation decisions, but I'm cheering on this PR in general!

I took a quick look at the example link above. Here are a few questions, which may or may not be premature at this stage:

• How does this feature interact with Neuroglancer's "linked to" and "colors linked to" layer setting? Is that straightforward, or will it be necessary to just disable that option for layers which make use of a color shader?

• Would it be possible to include the segment ID in the signature of segmentColor?

  vec3 segmentColor(vec3 color, bool hasProperties)

  I understand that it may not be simple due to the 32-bit limitations in glsl. But there are some cases where the segment ID encodes some meaningful information which could be visualized in the color map. In theory one could just add a segment property with that information, but there could be cases where the number of unique segments is huge and generating segment properties for all of them would be undesirable.

• It's cool that you can support tags. Just brainstorming here: Is it out of the question to also support string properties, at least for equality checks? Maybe something like this?

  if (prop("celltype") == propval("celltype", "Mi1")

[chrisj]

I'm not qualified to comment on the implementation decisions, but I'm cheering on this PR in general!

Thanks!

* How does this feature interact with Neuroglancer's "linked to" and "colors linked to" layer setting?  Is that straightforward, or will it be necessary to just disable that option for layers which make use of a color shader?

My assumption is that "colors linked to" should link the segment color shader as well. It currently doesn't so I'll work on that.

* Would it be possible to include the segment ID in the signature of `segmentColor`?
  I understand that it may not be simple due to the 32-bit limitations in glsl.  But there are _some_ cases where the segment ID encodes some meaningful information which could be visualized in the color map.  In theory one could just add a segment property with that information, but there could be cases where the number of unique segments is huge and generating segment properties for all of them would be undesirable.

Yes, there is a type uint64_t that already exists. In another branch I added a preprocess step to support using big ints directly in the shader (assuming it is small enough to fit in a uint64) using the n suffix. Which makes it easier to work with because otherwise, as you referred to, you have to split your segment id into a high and a low 32 bit values. The reason I left it out for now is that if you want a specific color for a specific id, the segment stated colors handles most of that unless you are trying to do something more complex. I forgot to update documentation to include the recently added isStated argument to support that.

* It's cool that you can support tags.  Just brainstorming here: Is it out of the question to also support `string` properties, at least for equality checks?  Maybe something like this?
  ```c
  if (prop("celltype") == propval("celltype", "Mi1")
  ```

String properties were unintentionally left out, we should be able to do that.

[chrisj]

segmentColorShaderControlState needs to way to differentiate between the lack of a segment property source and it not being loaded yet. I fixed it by waiting for layer.IsReady. If using isReady is the right way to handle it, I think the best fix is to convert loadingCounter and isReady to watchable/derivedwatchable values.

[chrisj]

I added support for string segment properties using this temporary syntax

stringPropertyEquals("celltype", "Mi1")
rather than
if (prop("celltype") == propval("celltype", "Mi1")

Just a little simpler to implement, especially when either the key or value are invalid since the result can just be replaced with false

I realized while working on string properties that segment property identifiers are not guaranteed to be unique. This color shader could just support the "first" property with the passed in identifier. I've been also thinking of how the invlerp control seems like the most natural way to interact with segment properties. We could add a property ui control. Something like

#uicontrol segprop prop1

You would then get a dropdown that listed all the segment properties:

• for numeric properties: prop1() would have the value just like current prop function.

• for tag properties: prop1() would a bool (or 0u or 1u)

• for string properties: a second dropdown would appear with the possible strings and based on that second selection, prop1() would act like a tag, (UPDATE: and/or string properties could get converted into an integer mapping to each unique string value which would still require string preprocessing for comparing values).

With this uicontrol, we could remove the new syntax I added which we do string preprocessing, it would also no longer require property identifiers to be unique to be fully supported.

[stuarteberg]

For whatever my opinion is worth, that all sounds great to me!

I like the idea of a dropdown. Tangentially, I think it would also be a great addition for other shaders, especially annotation shaders. Hopefully whatever you implement here will be conceptually similar (from a user's perspective) to whatever gets implemented for other shaders (if such a feature is implemented).

[chrisj]

Here is a deployment that shows the dropdown property uicontrol, I haven't implemented it for string properties yet

#uicontrol property highlight(type="tag")
#uicontrol invlerp gradient()
vec3 segmentColor(vec3 color, bool hasProperties, bool isStated) {
  if (!hasProperties) {
    return vec3(0.5, 0.5, 0.5);
  }
  if (highlight == 1u) {
    return vec3(0.0, 1.0, 0.0);
  }
  return colormapCubehelix(gradient());
}

https://cj-segment-color-dropdown-dot-neuroglancer-dot-seung-lab.ue.r.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B4e-9%2C%22m%22%5D%2C%22y%22:%5B4e-9%2C%22m%22%5D%2C%22z%22:%5B3.3e-8%2C%22m%22%5D%7D%2C%22position%22:%5B512381.78125%2C343507.8125%2C2600.5%5D%2C%22crossSectionScale%22:695.7566789495825%2C%22crossSectionDepth%22:-5.712090638488141%2C%22projectionScale%22:1048576%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22gs://h01-release/data/20210601/4nm_raw/%7Cneuroglancer-precomputed:%22%2C%22tab%22:%22source%22%2C%22name%22:%224nm_raw%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22gs://h01-release/data/20210601/c3/%7Cneuroglancer-precomputed:%22%2C%22subsources%22:%7B%22default%22:true%2C%22bounds%22:true%2C%22properties%22:true%7D%2C%22enableDefaultSubsources%22:false%7D%2C%22tab%22:%22rendering%22%2C%22segmentColorShader%22:%22#uicontrol%20property%20highlight%28type=%5C%22tag%5C%22%29%5Cn#uicontrol%20invlerp%20gradient%28%29%5Cnvec3%20segmentColor%28vec3%20color%2C%20bool%20hasProperties%2C%20bool%20isStated%29%20%7B%5Cn%20%20if%20%28%21hasProperties%29%20%7B%5Cn%20%20%20%20return%20vec3%280.5%2C%200.5%2C%200.5%29%3B%5Cn%20%20%7D%5Cn%20%20if%20%28highlight%20==%201u%29%20%7B%5Cn%20%20%20%20return%20vec3%280.0%2C%201.0%2C%200.0%29%3B%5Cn%20%20%7D%5Cn%20%20return%20colormapCubehelix%28gradient%28%29%29%3B%5Cn%7D%5Cn%22%2C%22segments%22:%5B%5D%2C%22shaderControls%22:%7B%22highlight%22:%7B%22type%22:%22tag%22%2C%22id%22:4%7D%2C%22gradient%22:%7B%22range%22:%5B0%2C0.5%5D%2C%22window%22:%5B0%2C0.5%5D%2C%22property%22:%22Sp%22%7D%7D%2C%22name%22:%22c3%22%7D%5D%2C%22selectedLayer%22:%7B%22size%22:528%2C%22visible%22:true%2C%22layer%22:%22c3%22%7D%2C%22layout%22:%22xy%22%7D

[chrisj]

Added support for string properties in the dropdown ui control

When you select a string property, you get a second dropdown. If you leave it at the default "any" state, the ui control value becomes the index shown the the right of the value in the dropdown. If you choose a specific value, it becomes 1u and every other value becomes 0u. You can see the behavior in the previous link. For that deployment, I forced a "debug_category" string property to be added to any loaded segment property source.

[fcollman]

is there a way to support stuart's ask for having a standalone string property that can be initialized with values from the shader code? or is that a seperate feature?