Masking Layer

[Lancelotbronner]

Is your feature request related to a problem? Please describe.

I'm creating overworld maps with a lot of different biomes and various terrain mixing patterns, too many in fact (over 10'000 variations).

In game I'm using full tiles and many layers for the terrain, with masks to mix them all together. I'd love to be able to have this in Tiled!

Describe the solution you'd like.

A mask layer would apply to all layers under it. The layers below would be rendered and then alpha multiplied with the mask's alpha (or using grayscale, whatever is easiest), which will allow one to mask layers!

I assume they would be placed in groups where they'd only apply to the layers in their group.

Describe alternatives you've considered.

As far as I know there are no alternatives, if there is one (that doesn't involve generating thousands of tilesets) please let me know!

[Lancelotbronner]

I've found Tiled 12.0 docs about blend modes, which I believe might fix my issue.
How can I get a preview build?

[zlago]

I've found Tiled 12.0 docs about blend modes, which I believe might fix my issue. How can I get a preview build?

have you looked in the actions tab? this is probably what you want https://github.com/mapeditor/tiled/actions/runs/22017312136

[Lancelotbronner]

I can't run any of them, I'm on an M1 macOS 26.

They don't even appear in Security & Privacy to allow them and even manually removing the quarantine attribute the app is still killed for codesign termination.

Trying to build locally I get

ERROR: Failed to start process launcher at '/opt/homebrew/libexec/qbs/qbs_processlauncher': Child process set up failed: execve: No such file or directory

yet I've brew install qbs so I don't know what else I can do.

[Lancelotbronner]

@bjorn I managed to get the nightly link you sent working! I was hoping to use blend modes for masking but it appears not to be possible, unless I missed something?

[Lancelotbronner]

Context: I'm using Tiled to generate images, not importing into a game engine so I actually need masking.

[Lancelotbronner]

For more context, my terrain and mask. The problem is I need variants for the terrain else I'd have made separate tilesets for each.

[eishiya]

This should be doable with the "destination in" composition mode on the "mask", but it looks like Tiled doesn't expose that as a blending mode even though it is available in Qt.
However, even if that's exposed, for this to actually be useful, Tiled would probably need to first implement group-aware blending, which was discussed in #3932 but I don't believe it was implemented because it would probably require bigger changes to Tiled's renderer (and may similarly be challenging to implement in many game engines, requiring a lot of extra compositing). The discussion there was about composing a group first and then blending the group, which is a little different from what's needed for masks, which is restricting the "destination" to be only the masked layer (or group of layers).

Rather than their own layer type, I think it would be preferable to make a "Mask" blend mode, or maybe "Mask (Alpha)" and "Mask (B&W)", the latter would use the average value as the alpha instead of the alpha channel, making it possible to use greyscale tiles as masks. Unlike the other blend modes, "Mask" would only take the layer immediately below it as the "input" (to mask multiple layers at once, they could be grouped). This would allow easily "converting" a mask to a plain tile layer for better visibility while editing, and make it clear that the layer is edited exactly the same as any other tile layer (including with Terrains, Automapping, etc). But, due to the different way in which this blend mode is applied, it might be good to treat these layers as somewhat special, perhaps by changing the layer's icon when the blending mode is set to a mask mode, and/or showing some sort of visual link between the mask and the layer/group is affects.

[Lancelotbronner]

I'd be satisfied with the absolute simplest POC -- I can workaround the rest!

I'm also volunteer to help test it if someone wants to work on it!

[bjorn]

Indeed, as explained by @eishiya such a feature would require a rather big change to the rendering and is also something I'd schedule after a modernization of the renderer because it would be quite slow in software. So thanks for creating an issue about it, but it's out of scope for now.

As a workaround, if you need to work with masked terrain tiles now, you could pre-mask them in an image editor and use the resulting images in Tiled. Realize this would lead to rather a lot of images.

[Lancelotbronner]

Thanks for your time investigating this! I'll go ahead with the workaround (if it's feasible) for now.

The game (Dwarf Fortress) is actually using a software renderer for this.

[eishiya]

Generating the composite tilesets should be something you can automate in a scriptable image editor like Photoshop, Krita, or Aseprite (and probably ImageMagick too, but I'm not familiar with that). Tiled scripting can also be used to generate these composite tileset images. In theory you could even use Tiled scripting to automatically "convert" between "masked" maps and maps that use the workaround of pre-masked tiles, so that you'd only need the composite tilesets for viewing the results in Tiled, but could use the original tilesets + masks in-engine, avoiding the bloat of all those composite tilesets.

It should also be possible to set up a Tiled script that lets you edit using the original tiles (including the mask tiles) in Tiled, but preview the results directly in Tiled, e.g. by creating an Image Layer and software-rendering the composite results to that layer. That would be less smooth than doing it with pre-composed tilesets, but may be more convenient since you would be able to edit via the masks and may be easier to write if you're more familiar with software rendering than with the Tiled's TileMap/TileSet API.

[Lancelotbronner]

Since I'm not familiar with either of these and my deadline is closing up (I'm making a D&D world map in the Dwarf Fortress universe using their assets, not making a game!), I've started doing it manually.

It's very nice to know that an automated option might be possible!