Implement constant LOD glTF extension

[danielzhong]

Description

This PR adds support for the new EXT_textureInfo_constant_lod glTF extension. See the extension spec PR CesiumGS/glTF#92 which is in the final stages of review.

Constant level-of-detail ("LOD") is a technique of texture coordinate generation which dynamically calculates texture coordinates to maintain a consistent texel-to-pixel ratio on screen, regardless of camera distance. As the camera zooms in or out, the texture coordinates are recalculated so that the texture pattern remains at approximately the same visual scale. The transition between scale levels is smoothly blended to avoid abrupt changes. The general use case of this is to create textures that look good at different zoom levels, commonly for surfaces like grass or gravel on a ground plane in large models.

See the spec README for detailed info about the formulas used.

Notes:

• While according to the spec the extension can be present on any textureInfo property, this implementation only supports it on baseColorTexture and normalTexture to a limited extent.
  • The normal map texture limitation is that if a material has both a base color texture and a normal texture that use the extension, the base color texture's constant LOD properties (repetitions, offset, etc.) will also be applied to the normal texture and override the normal texture's properties if they are different.
  • This is done to match the iTwin.js implementation, and the intention is to keep the textures in sync as the texture coordinate blending/recalculation is done.
• The feature is not currently supported for emissiveTexture, occlusionTexture, or any other textureInfo

Model with EXT_textureInfo_constant_lod grass texture very zoomed in:

Same model zoomed out. Note how the texture remains at the same visual scale:

Gif from iTwin.js to illustrate the effect:

Issue number and link

#12892

Testing plan

I added a new dev Sandcastle to test this feature for two different base color textures, as well as a normal map. See the test models in Specs/Data/Models/glTF-2.0/ConstantLod/gltf/. I also added some unit tests to GltfLoaderSpec.js.

Test models free texture sources:

• Checker texture
• Grass texture
• Normal map texture

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
• I have updated the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[github-actions]

Thank you for the pull request, @danielzhong!

✅ We can confirm we have a CLA on file for you.

[eringram]

@javagl would you be able to review this PR, given you might have some context from reading the spec? Or possibly @lilleyse as this is to support a glTF extension for design models. No specific deadline or urgency, but would be great to get some eyes on it this week

[javagl]

I did not yet thoroughly read the spec, but will have a closer look (and at least try out this PR) soon. I'm sure Sean can provide more profound feedback about things like best practices for this, and the implementation approach in general.

[javagl]

Some "minor/small" things, and a few high-level comments for now:

---

The
Specs/Data/Models/glTF-2.0/ConstantLod/gltf/checker_txture.png
is actually a WEBP file (wrong file extension).
(This causes a validation error because of the mismatch between the MIME type and the actual type, and because WEBP would require an extension)

(There's an archive attached blow that contains the PNG version)

---

The ModelComponents constantLod is documented to be
@type {object}
but must be
@type {object|undefined}

---

More generally: Several functions in the MaterialPipelineStage now receive RenderResources as a parameter. This parameter is usually not documented. In the cases where it is documented (e.g. in processConstantLod), it is typed as
@param {PrimitiveRenderResources}
but should be
@param {PrimitiveRenderResources|undefined}
(Admittedly, some parameters (like the "default texture") are generally not documented, *sigh*... just mentioning it...)

---

While according to the spec the extension can be present on any textureInfo property, this implementation only supports it on baseColorTexture and normalTexture to a limited extent.

I understand the reasoning behind that. Supporting it everywhere could be a lot of effort. Roughly speaking: Every call to processTexture would need the renderResources to be passed in (raising the question of whether they can still be undefined in the JSDoc). This would likely affect shader in many places as well....

Not in the scope of this PR, but ...

...out of curiosity, I created an example where both the base color and the emissive texture contain the extension:

ConstantLod_CheckerEmissive.zip

(This contains the checker texure as a PNG)

It works within the constraints that are established here (i.e. only for the base color):

(Why does the emissive texture appear to be rotated by -90 degrees? Haven't investigated that yet...)

The normal map texture limitation is ...[...]

[...] to match the iTwin.js implementation

Again, I see the reasoning behind that. How a "full, proper" support could be implemented is probably beyond the scope of this PR. But one could still consider to track the current limitations in an issue (maybe after/when this PR is merged)

---

I could imagine that there are some tricky considerations for the cases where this extension is supposed to be combined with KHR_texture_transform. If it is not allowed, then this should be mentioned in the specification. If it is allowed, I could create some test asset for that.

---

I might do another pass, depending on the responses and the desired level of nitpicking strictness for the review.

[eringram]

@javagl
Thanks, I fixed the WEBP file and incorrect docs you mentioned.

(Why does the emissive texture appear to be rotated by -90 degrees? Haven't investigated that yet...)

I'm confused by this as well and took a look to see if the orientation of the emissive texture was changed when I removed the extension. Here's the model as-is vs. when I remove this extension from both textures (base color & emissive). I think this indicates a bug with the orientation of the UVs produced by the constant LOD shaders?

Constant LOD	No constant LOD

I could imagine that there are some tricky considerations for the cases where this extension is supposed to be combined with KHR_texture_transform. If it is not allowed, then this should be mentioned in the specification. If it is allowed, I could create some test asset for that.

I don't in theory see a problem with them being combined... I just did a quick test where I added the following to the ConstantLod_Checker.gltf's baseColorTexture that already has constant LOD:

"KHR_texture_transform": {
  "offset": [0.1, 0.1],
  "scale": [2.0, 2.0],
  "rotation": 0.75
}

and it seems like KHR_texture_transform is completely ignored when constant LOD is also present. This is because the UVs calculated from texture transform on this line aren't used in the constant LOD calculation a few lines later. One solution I can think of is applying the KHR texture transform to the constant LOD coords tc1 and tc2 here before they are mixed.

[javagl]

if the orientation of the emissive texture was changed when I removed the extension.
...
I think this indicates a bug with the orientation of the UVs produced by the constant LOD shaders?

I probably should have said that more explicitly: I think that this is not related to this PR. (It may be worth examining that further, and maybe opening a dedicated issue - and it's quite odd that this seems to happen only for the emissive texture (until now) - I'll take a TODO on my list to check this, for emissive, occlusion, etc., and maybe open an issue (or PR) eventually)

and it seems like KHR_texture_transform is completely ignored when constant LOD is also present.

I mentioned "tricky considerations", but the first one is relatively easy: Should it be possible to combine them (in a form where they both have the intended effect)?

If the answer to that is "No", then one could simply say in the spec: "This extension cannot be combined with KHR_texture_transform" (and be done).

If the answer is "Yes, it should be possble to combine them", then the tricky considerations come in - namely, about the where and how. I have not yet thought that through. The second place that you linked to looks like one that could make sense - i.e. transforming the tc1 and tc2 with a mat3 textureTransformMatrix that could be passed in there. This mat3 would have to be computed in the same way as it is done for other texture lookups, and be the identity matrix by default, but then people will scream "Oh dear, the matrix is computed twice, and usually an identity (without an effect)" - and then you'll have to insert another set of #ifdefs, because of (jazz hands:) performance.

Maybe the answer to the first question is 'No'...?

[eringram]

I probably should have said that more explicitly: I think that this is not related to this PR. (It may be worth examining that further, and maybe opening a dedicated issue - and it's quite odd that this seems to happen only for the emissive texture (until now) - I'll take a TODO on my list to check this, for emissive, occlusion, etc., and maybe open an issue (or PR) eventually)

Based on the screenshots I included, is the fact that the base color texture is a different orientation w/ and w/o this new extension not a concern? that's what made me think it's related to this PR, but I'm not familiar with any existing problems with UVs so could be wrong. Here's a clearer comparison of what I mean:

Constant LOD	Emissive texture, no CLOD	No emissive, no CLOD

(See the orientation of the numbers and letters in the texture)
So I guess in other words, I suspect the emissive texture is correct and CLOD shader is making the base color texture incorrect, because while it has offset and repetitions props, it shouldn't rotate anything.

Maybe the answer to the first question is 'No'...?

I feel like the answer is "yes it should be possible, but out of the scope of this issue since it's not a common/high priority use case"-- I'm hesitant to explicitly say it's not allowed in the spec, but maybe we don't have to spend the time on implementing it yet since it hasn't come up until now when discussing design models that use EXT_textureInfo_constant_lod. And we can document in the code and an issue that they are currently mutually exclusive in CesiumJS.

@weegeekps do you think this is fair on the topic of combining EXT_textureInfo_constant_lod and KHR_texture_transform?

[javagl]

Ahhh 💡 ... sorry, I completely missed that. (For some reason, I was only looking at that "Emissive" string and thought "Well, it's 'wrong' in both cases?!?")

But now I see: Yes, apparently the extension "rotates" the texture coordinates, and that should not be the case. I said "rotates" in quotes, because it might simply be a mixup of u and v (given that they are likey both in [0,1], that could easily explain it).

Quickly skimming over the code: There's that constantWorldToEnu and czm_model at https://github.com/CesiumGS/cesium/pull/13121/changes#diff-1fb9ea4da146861f0b9c81c8dc1e5ed29aba51dcc3b772f57044800c267f85faR9 that may play a role here. There are different conventions for the "up-axis" in glTF and 3D Tiles, and CesiumJS has to deal with both of them, which is an endless source of confusion. Depending on which coordinate conventions these matrices are using, this seems like a likely source of errors (but that's just a pointer - I didn't do the math yet, and I don't say that the problem is there, only that I'd start looking there...)

[eringram]

After discussion with @markschlosseratbentley (and his discussion with @weegeekps), this is a summary of remaining issues on this PR that I am working on:

1. Fixing the "rotated" coordinates bug and
2. Investigating how feasible it is to support both EXT_textureInfo_constant_lod and KHR_texture_transform together
   1. Current behavior: if both are present, EXT_textureInfo_constant_lod UV calculations completely override KHR_texture_transform (maybe other undefined behavior as well)
   2. Since CLOD is an EXT_ prefixed extension, we want it to be compatible with other common extensions. So if it's not feasible to support both (i.e. seems like a large time investment), it may be changed to BENTLEY_ in the spec.

[eringram]

1. I've fixed the UV bug by swapping the X and Y in the vertex shader (and removing a subsequent axis flip), it seems like there was a mix-up between which was expected to be East/North from enuDir. New screenshots:

Constant LOD	Emissive texture, no CLOD	No emissive, no CLOD

---

2. On compatibility with KHR_texture_transform:
   • I added a new overload function of constantLodTextureLookup in the fragment shader that takes a 3rd argument, textureTransform. When both EXT_textureInfo_constant_lod and KHR_texture_transform are present, this overload is used and czm_computeTextureTransform(tc1, textureTransform) is applied to the CLOD texture coordinates.
   • Added czm_computeTextureTransform as a new shader built-in to replace computeTextureTransform, which was previously defined in MaterialStageFS.glsl. I added this because in the case of CLOD being used only for base color, the CLOD shader didn't have access to the computeTextureTransform definition in MaterialStageFS.glsl. So this way the function is now accessible anywhere these built-ins are, and doesn't have to be redefined in ConstantLodStageFS.glsl.
   • Added glTF models for these new test cases with emissive texture, transform, and normal map with transform. Added because they were useful to test edge cases with, but lmk if it's excessive to include all of them.

Constant LOD alone	Texture transform alone	Both texture transform & CLOD

Also tested normal map support for both texture transform and CLOD. I added a normal map that matches the checkerboard color texture so it would be easy to see if they are aligned.

The difference between these images is a little hard to see, but the normal map makes the right image's letters and some square borders appear darker/more pronounced. You can tell both textures are transformed since they are aligned.

Both texture transform & CLOD, no normals	Both texture transform & CLOD w/ normal map

[eringram]

@javagl Adam gave the interaction with KHR_texture_transform in this implementation his approval and said the final review goes to the Cesium team. Would you be able to take another pass at the code?