app,gpu,op: [android,ios,macos,windows] add ExternalOp paint for external views

[inkeliz]

This change adds paint.ExternalOp, which allows native views, activities, or HWNDs to appear "inside" a Gio layout.

The implementation uses a punch-through method. Gio marks a region as transparent so the layer below becomes visible. One user-defined function receives the area/position and renders there. Before this change, integrating external components such as WebView, Ads, or Camera views had two problems.

1. Gio content could not render above the external view.
2. The external view required an absolute position. That was difficult when combined with offsets or transformations.

The paint.ExternalOp reduces both issues while keeping the API simple. The operation includes a callback that runs every frame. The callback can update position, size, and z-order to keep the native view synchronized with the Gio layout. Platform behavior required small adjustments, mainly on Windows.

Windows uses an extra child window:
MainWindow → GioWindow

This structure allows additional HWNDs to attach to the MainWindow while GioWindow remains on top. For example: MainWindow → GioWindow → WebView

ExternalOp exposes a region of the GioWindow so the underlying HWND, such as a WebView, becomes visible in that area.

On macOS, iOS, and Android the rendering stack already uses additional layers. No structural change was required there, only the background was changed to Transparent.

---

What the ExternalOp do:

1. Make the current clip-area transparent (it will show what is below the Gio layer, in z-index).
2. Ignore events (like clicks and touch) in that clip-area (at OS level, no Gio proxying).
3. Call a user-defined function, which is responsible to render the content in that area.

[inkeliz]

Note that the "punch-through" has some issues, the synchronisation is not perfect. I'm trying to address that, but I don't think it's possible to fix the issue.

The other method is off-screen rendering to a OpenGL-ish buffer, and render as an image. That would be always in-sync with Gio, but the pointers/clicks (and other events) becomes an issue.

Anyway, it's possible to add other method to ExternalOp in the future, if anyone wants. The current one is the easiest.

---

Also, the area is always rectangular. I mean, custom paths or circular will not work properly. That is a big limitation, but better than nothing.

[inkeliz]

Maybe I need to add "https://todo.sr.ht/~eliasnaur/gio/428" as reference in the commit.

[inkeliz]

I submit to SourceHut to run all tests.

[ddkwork]

abs pos see this

ddkwork@3e5e590

[eliasnaur]

Nice work.

Note that the "punch-through" has some issues, the synchronisation is not perfect. I'm trying to address that, but I don't think it's possible to fix the issue.

What makes perfection impossible? Is it the approach ("punch-through") or is the issue the way Gio is currently designed?

The other method is off-screen rendering to a OpenGL-ish buffer, and render as an image. That would be always in-sync with Gio, but the pointers/clicks (and other events) becomes an issue.

FWIW, I agree that the punching through and letting the external UI content render itself is the most flexible approach, in particular because of event handling.

[inkeliz]

What makes perfection impossible? Is it the approach ("punch-through") or is the issue the way Gio is currently designed?

Timing. The current code do:

• Swap framebuffers/render new frame
• Call ExternalOp callback to adjust position

Maybe we could do the opposite, but optimally we need that to happen closest as possible. If we do the opposite, the view might change while the old frame is visible.

I'm not sure if exists a native way of "double buffering native views".

---

Some "views" (like WebView) are quite slow to resize (see tauri-apps/tauri#6322). Changing the size/offset has the same issue. That is not a Gio issue, but how the external view will react.

[eliasnaur]

What makes perfection impossible? Is it the approach ("punch-through") or is the issue the way Gio is currently designed?

Timing. The current code do:

* Swap framebuffers/render new frame

* Call ExternalOp callback to adjust position

Maybe we could do the opposite, but optimally we need that to happen closest as possible. If we do the opposite, the view might change while the old frame is visible.

I think it would be a mistake to expose an API that makes seamless integration impossible. Resizing the window comes to mind where Gio's macOS window content would not update in sync with the rest of the window, leading to ugly artifacts.

macOS has Core Animation transactions, and I think Wayland has something similar. Will your design allow the Gio content and one or more external views (layers, HWNDs etc...) to be explicitly synchronized?

What about the design where Gio is given the external contents as a handle (UILayer, HWND etc.)? Just like Gio's various ViewEvents? That way, Gio can explicitly position and arrange redraw at the same time the Gio content redraws.

[inkeliz]

macOS has Core Animation transactions, and I think Wayland has something similar. Will your design allow the Gio content and one or more external views (layers, HWNDs etc...) to be explicitly synchronized?

Currently, it doesn't. Maybe extend ExternalOp to provide another method in the future?!

What about the design where Gio is given the external contents as a handle (UILayer, HWND etc.)? Just like Gio's various ViewEvents? That way, Gio can explicitly position and arrange redraw at the same time the Gio content redraws.

I tried to "keep it simple", by calling a Go code instead of messing with (Layer, HWND, Surface...) in Gio itself.

I'm searching how can I move things atomically. On Android we have "SurfaceTransaction" (https://developer.android.com/ndk/reference/group/native-activity#asurfacetransaction_setposition), but it's limited to API 31 (or API 33, because the Java version documentation is 33). So, that is quite recent too. The SurfaceControlViewHost is also from API 30.

EDIT: I don't know how to do that targeting Android <29. Far I remember, from Flutter, it uses the texture-hack for old Android. The "callback" might be easier solution for old Android.

---

This change is already ~600 lines. Most of the code (from GPU and such) would remain the same, either using GoCallback or another LayerComposition. In both cases we need the position and render the transparent layer. In both cases we need to ignore events.

Maybe we can split it into two phases:

• Phase 1: Callback-only = NewExternalOp(callback func(visibleArea, totalArea, zIndex))
• Phase 2: Callback+View = NewExternalOp(callback func(visibleArea, totalArea, zIndex), getLayer func() uintptr)

Gio will use the given layer when possible. If we cannot use the given layer (layer == nil, on old Android), we invoke the callback method (if the callback is not nil).

[CoyAce]

This PR is worth supporting. With ExternalOp, users have the freedom to choose whether to use it or not. Currently, users don’t have that choice and have to implement it themselves. It supports completing the blueprint in a piecemeal way, rather than aiming for a utopian, one-shot solution.

[CoyAce]

This aligns with Karl Popper's philosophical critique of utopian engineering. He elaborates on the distinction between these two approaches in The Open Society and Its Enemies.

[eliasnaur]

I didn't mean to imply that @inkeliz should implement atomic updating of Gio and external content in this PR. What I mean is that (1) we should have a single API for embedding external content and (2) the API should support atomic updating, on a best effort basis (depending on what we've implemented, OS version etc.).

Atomic updates may seem like a small detail, but it's one of those issues that to me makes the difference between smooth and janky. A recent example is when I fixed window resizing on macOS to synchronize with the contents.

[inkeliz]

What do you think about changing:

func NewExternalOp(callback func(visibleArea, elementArea image.Rectangle, zIndex int)) ExternalOp {
	return ExternalOp{
		callback: callback,
		handle:   new(int),
	}
}

To:

func NewExternalOp(callback func(visibleArea, elementArea image.Rectangle, zIndex int), getView func() uintptr) ExternalOp {
	return ExternalOp{
		callback: callback,
		getView: getView,
		handle:   new(int),
	}
}

In my opinion that allows Gio to move the View (if getView returns a valid view) while keeping the callback as a fallback.

Maybe use an interface instead?

type ExternalView interface {
  Show(visibleArea image.Rectangle, elementArea image.Rectangle, zIndex int)
  View() uintptr
}

Or.... Another alternative ?

[CoyAce]

In my view, there are two core issues here: first, Gio needs to support atomic updates; second, ExternalOp should provide a native view handle to leverage Gio's atomic update API for synchronized rendering. Simply providing a handle without Gio's atomic update capability is meaningless, and therefore the relevant interface should not be exposed at this stage. On a positive note, the current discussion around atomic updates points in a clear direction for future architectural evolution.

[CoyAce]

From an architectural standpoint, the handle + atomic update model is clearly the more elegant long-term direction. That said, callback-based integration still holds value—both as a compatibility fallback and as a lower-friction entry point for simpler use cases. Critically, these two approaches should not be coupled within the same API surface. Callback and handle ought to be separate implementations of ExternalOp, not merely two modes of a single implementation. This ensures cleaner abstraction, better testability, and future extensibility without accumulating technical debt.

[eliasnaur]

In my view, there are two core issues here: first, Gio needs to support atomic updates; second, ExternalOp should provide a native view handle to leverage Gio's atomic update API for synchronized rendering. Simply providing a handle without Gio's atomic update capability is meaningless, and therefore the relevant interface should not be exposed at this stage. On a positive note, the current discussion around atomic updates points in a clear direction for future architectural evolution.

Can you explain what's gained by also offering the callback API, other than it's easier to implement? Exposing the atomic-capable API now, even before we have atomic updates, is the right decision: users won't have to change their programs when we implement atomic updates.

Or.... Another alternative ?

I imagine something like

package paint // import "op/paint"

// EmbedOp embeds an external view at the origin position of
// the current transform. See package app for ways to construct
// EmbedOps.
type EmbedOp struct {
    handle uintptr // or an array of uintptrs, as needed
}

func (o *EmbedOp) Add(o *op.Ops)

and

//go:build darwin
package app

// AppKitView represents an AppKit UIView.
type AppKitUIView struct {
   Layer uintptr
}

func (v *AppKitUIView) Op() paint.EmbedOp

[CoyAce]

Hi Elias, thanks for the thoughtful question.

You're right that exposing an atomic-capable API now would let users adopt it early and avoid breaking changes later—that's a valid forward-looking concern.

After more thought, I realized the tradeoff is more nuanced.

The real tradeoff: who carries the complexity?

A callback API might seem simpler at first glance, but it actually shifts complexity from Gio to the user:

// Callback: user writes this
NewExternalOp(func(visibleArea, elementArea image.Rectangle, zIndex int){
    // User is now responsible for:
    // - Holding the native view reference
    // - Calling platform-specific APIs
    view:=viewFor(handle)
    view.MoveTo(visibleArea,zIndex) // This line hides immense complexity
}).Add(ops)

A handle-based API, by contrast, keeps complexity inside Gio:

// Handle: user writes this
area:=clip.rect(rect).push()
paint.EmbedOp{handle}.Add(ops)
area.pop()
// gio internally handle like this
view :=  viewFor(handle)
view.MoveTo(rect) // Gio handles all platform details internally

// Gio internally does the hard work:
// - JNI/FFI calls
// - Error handling

Why this matters

For most developers, in most scenarios, they don't want to know about platform details.

They just want their WebView to appear in the right place. Callback forces them to care about platform details; handle-based API lets them not care.

What atomic updates change—and don't change

Atomic updates make handle-based APIs more powerful by adding precise timing. But they don't change the fundamental tradeoff: callback still offloads complexity to users, handle still encapsulates it.

My current view

Rather than viewing callback as a "temporary fallback" and handle as the "real future," I see them as representing a fundamental design choice:

• Callback: Gio says "here's the position, you figure out the rest"
• Handle: Gio says "give me the view, I'll take care of it"

I believe the latter is more aligned with Gio's philosophy of providing a high-level, cross-platform abstraction. Users who need low-level control can always access the underlying handle directly—but for the 80% case, they shouldn't have to.

Curious to hear your thoughts.

[inkeliz]

Can you explain what's gained by also offering the callback API, other than it's easier to implement?

I mean it's easier to implement. :P

One example is WebView2 on Windows. It needs to call put_bounds (see https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2host?view=webview2-0.9.430#put_bounds).

That call can't be done by Gio itself. Gio will only move/resize the sub-window. The implementer of WebView2 will need to handle and proxy all WM_SIZE messages (and similar) to call WebView2 functions.

In other words:

1. Gio -> Callback -> (External Code: Calls PutBounds)
2. Gio -> Move HWND -> (External Code: Receive WM_SIZE -> Calls PutBounds)

Well, that is the same "easier" argument. But, it can be easier on the other side too (not only in-Gio).

---

I'll update try to update my code (and my WebView implementation test) to something similar to @eliasnaur suggestion:

type AppKitUIView struct {
   Layer uintptr
}

type WindowsView struct {
   HWND uintptr
}

type AndroidView struct {
   View uintptr
   SurfaceControl uintptr
}

I'll not implement atomicity. Only the resize and movement and clicks/touch coords translation.

---

Note: I'm using WebView as example because it's one of the most complex one (it needs to handle events, inputs, display stuff...)

[inkeliz]

I'm currently testing it on a couple of Android, on my iPhone and macOS. The Windows seems to be the most difficult one to integrate with external stuff. The iOS/macOS are almost lag-free, not sure if it's the API or the CPU/GPU performance. _On Android I'm testing with couple of old and low-end devices, so even native apps are slow. _

EDIT: The Android version is laggy even on "high-end" devices (like Motorola Razr 50 Ultra and Google Pixel 9).

[inkeliz]

One way to mitigate the "laggy" feeling is setting the background to be the same of the rendered area. I'm not sure if it's worth adding it and I'm not sure how to do that.

Imagine that you have a Yellow box (100% of the screen) and in the middle you have a VideoPlayer (EmbedOp). If the "root background" is Yellow, any "gap" (between Gio and VideoPlay) would be Yellow. Making it less noticeable. The "gap" can happen during animation.

---

Since this patch is already large enough, I'll keep it as simple as possible (without such mitigation). That can be useful for old Android (without atomic alternatives).

[inkeliz]

While testing on Android I noticed one issue: Unisoc.

Sounds like multiple Unisoc devices (like SPC Discovery SE, with Unisoc Tiger T310) doesn't produce a texture transparent. In other words: the "EmbedOp"-area is black, not transparent.

The workaround is moving away from SurfaceView and use TextureView. I don't think it's the right way of fixing it.

---

That is the list of devices that I use for testing:

Motorola | Moto G 2nd Gen
Motorola | Moto E6s
Motorola | Moto E14
Motorola | Moto E15
Motorola | Moto G35 5G
Motorola | Moto G 2nd Gen
Xiaomi | Redmi 7A
Xiaomi | Redmi Note 9
Xiaomi | Redmi A3
Xiaomi | Redmi A5
Xiaomi | Redmi Note 12 5G
Xiaomi | Redmi Note 14
Xiaomi | POCO M7 Pro 5G
Samsung | Galaxy A04
Samsung | Galaxy A06
Samsung | Galaxy A13
Samsung | Galaxy A26 5G
Samsung | Galaxy A20e
Samsung | Galaxy A21s
SPC | Discovery SE
Realme | Note 70T
Google | Pixel 9
Huawei | P8 Lite (2017)
Tecno | Spark Go BG6
Vivo | Y22S
Vivo | Y16

[CoyAce]

How about detecting the hardware manufacturer and selecting the appropriate view type based on the hardware? For example:

import android.os.Build;

public static boolean isUnisocDevice() {
    // API 31+ (Android 12+) has SOC_MANUFACTURER publicly available
    if (Build.VERSION.SDK_INT >= 31) {
        String socManufacturer = Build.SOC_MANUFACTURER;
        return "UNISOC".equalsIgnoreCase(socManufacturer) ||
               "Spreadtrum".equalsIgnoreCase(socManufacturer);
    }
    
    // API < 31 fallback to hardware name detection
    String hardware = Build.HARDWARE.toLowerCase();
    return hardware.contains("ums") ||      // Unisoc chip code prefix
           hardware.contains("unisoc") ||
           hardware.contains("spreadtrum");
}

[CoyAce]

I looked into it. The SurfaceControl API is the officially recommended way to manipulate layers on Android. However, it requires Android 10+, and it's still uncertain whether Unisoc's implementation is reliable. If that approach doesn't work out, we'll ultimately have to fall back to TextureView.

[inkeliz]

I don’t think “blacklisting” is a good option. Some devices, like the Motorola E14, use Unisoc and work, while others, such as the Xiaomi A3x and SPC Discovery SE, do not. Also, these devices are much slower. On AnTuTu, the difference between the Motorola and SPC is about 30,000 points, but the Gio app is much slower on the SPC.

I will look at how Flutter works (https://github.com/flutter/flutter/blob/master/docs/platforms/android/Texture-Layer-Hybrid-Composition.md).

I will investigate which backend is being used and whether some flag or extension is not available. If there is a way to query the GPU and detect that a required feature is not supported, that would be better than blocking devices based on the GPU name.

[CoyAce]

Alright, this is indeed a better practice of pursuing excellence. Looking forward to your good news.