feat(worker): opt-in container-image runtime for GPU apps (single-machine)

[nilsmechtel]

Summary

Opt-in container-image-as-Ray-Serve-runtime for BioEngine GPU apps, single-machine mode only. An app can declare @bioengine.app(container_image=…) and its replica runs inside that prebuilt image instead of the default pip/py_modules runtime — for heavy or conflicting dependency stacks that are better shipped as an image than resolved per-actor.

Pip/py_modules stays the recommended default. The whole path is gated behind the new worker flag --enable-container-runtime (off by default) plus the per-app container_image= param. With the flag off, nothing here changes worker behaviour, and the KubeRay/multi-node path is untouched.

Proven end-to-end on a throwaway rootful single-machine worker: deploy_app → HEALTHY, version_verified=True, GPU probe returns a real CUDA context (torch compute on an RTX 3090, not an NVML false-positive).

What's in the PR

Framework (bioengine/)

• _app/decorators.py — container_image= param; drops a {"container": {"image": …}} marker into runtime_env. Raises ValueError if combined with pip (Ray forbids mixing a container runtime_env with pip/py_modules).
• _app/bootstrap.py — a container branch in build_and_run_application builds a minimal runtime_env ({container, env_vars} — the only keys Ray permits alongside container) and returns before the normal source/hook injection. The image is the sole source of code (bioengine + app source baked on PYTHONPATH). GPU wiring rides in podman run_options: --device nvidia.com/gpu=all (when GPU requested) + LD_LIBRARY_PATH to the driver libs.
• cluster/ray_cluster.py — when enabled, generate a podman-4.9.3-compatible CDI spec at startup (nvidia-ctk, --disable-hook update-ldcache, down-version to cdiVersion 0.6.0, strip additionalGids + its orphaned bare-int list items) and export the driver-lib dir for the replica LD_LIBRARY_PATH. No-op with a warning if not root or nvidia-ctk is absent.
• worker/__main__.py — --enable-container-runtime flag (Ray Cluster group; flows straight into RayCluster via ray_cluster_config).

Image (docker/)

• worker-container-runtime.Dockerfile — a dedicated rootful worker image (ubuntu:24.04 for podman 4.9.3, nvidia-container-toolkit, a build-time Ray patch dropping the hardcoded --userns=keep-id flag, guarded to fail the build loudly if a Ray bump renames the anchor).

Design notes

• Why a separate image, not a python:3.11-slim variant. The ubuntu:24.04 base is forced by the podman 4.9.3 pin (podman 5.x rejects --pid=host on cgroup-v1 hosts; Debian slim's apt has neither 4.9.3). The prod worker stays lean on python:3.11-slim — its smaller CVE surface shouldn't be regressed to serve a niche opt-in path.
• Packaging (productization, not in this PR). Intended to publish as a separate GHCR package bioengine-worker-container-runtime:X.Y.Z sharing the worker version, so the heavy image's scan findings stay out of the bioengine-worker package. The CI build+tag step is deferred.

Scope / not included

• The PoC CUDA app, its image, and the throwaway compose are kept local (not committed).
• Deferred: auto build-on-top pipeline, a manifest runtime.container_image field, docs/skill updates, CI publish wiring, KubeRay/multi-node.

Status

Draft — dev-image validation + version bump before marking ready.