core: fix background screenshot on nvidia

[PointerDilemma]

Fixes screencopy on nvidia by falling back to a shm buffer when dma failes.

Closes #59

[PointerDilemma]

I guess we probably want a option to enable it per default, so that we don't have to wait for dma to fail.

[PointerDilemma]

I tested with a 3060ti and 550.142 drivers on Hyprland (10-bit as well) and sway (sends a different shm format, tested 24->32).

[vaxerski]

generally dma on nvidia should work and there's likely a mistake in the code somewhere. I mean, aquamarine creates buffers just fine.

I guess this is a fair enough compromise for now though (shm is slooow)

[PointerDilemma]

you mean it should work with non-linear modifiers??

[vaxerski]

yeah, nvidia can render to non-linear.

[PointerDilemma]

... but it can't bind them to GL_TEXTURE_2D right?
And when using GL_TEXTURE_EXTERNAL_OES, i do get a frame for non linear mods and can render it, but there are graphical glitches, that i assumed to be cause egl cannot handle non linear. Did i get smth wrong here?

[vaxerski]

no, external is needed for linear. Most modifiers should be fine with GL_TEXTURE_2D. See
https://github.com/hyprwm/Hyprland/blob/d335c8f10135e165a1085fd14134c659e4caf05a/src/render/OpenGL.cpp#L379-L422

[PointerDilemma]

Using eglCreateDRMImageMESA instead of eglCreateImage works. Probably because the buffer get's exported via mesa and not nvidia.

[PointerDilemma]

Ok since dma also works for nvidia now, I removed the auto option from general:screencopy_mode.
We will keep the cpu buffer implementation just in case someone needs it.

[PointerDilemma]

Ok so on my old intel laptop eglCreateDRMImageMESA did not work. presumably because the frame was not exported by mesa but by an intel driver??

It now tries eglCreateImage if the mesa one failed to fix that.

[PointerDilemma]

So in the end we were just missing EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT and EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT. No need for eglCreateDRMImageMesa.
Thankfully vaxry pointed me to the aquamarine source, where I noticed that was missing.

[vaxerski]

wonderful