[RFC] Ship prebuilt C++ runtime and delegate binaries in the ExecuTorch pip wheel

[shoumikhin]

TL;DR

Today you cannot use ExecuTorch's C++ runtime, CPU kernels, or a hardware
delegate (like CUDA) from a pip install alone. You have to clone the repo,
sync submodules, and build from source with CMake. This proposal is to bundle
prebuilt binaries in the wheel, the same way PyTorch ships libtorch and CUDA
wheels, so that pip install gives you a runtime you can link against and run.
As a rough design-time estimate the base runtime would add a small amount (order
of a fraction of a megabyte) and a full CUDA build would land in the mid-teens of
megabytes, but exact sizes are a release-gate output, not a commitment.

This issue is for gathering feedback on the approach before any implementation.

The problem

ExecuTorch has two halves:

1. The "ahead of time" (AOT) half in Python: export a model to a .pte file.
2. The runtime half in C++: load a .pte and run it, optionally on a hardware
   backend (a "delegate") such as CUDA, XNNPACK, or Core ML.

The pip wheel today ships the Python/AOT half plus the pybind extension. For a
standalone C++ app it does NOT provide:

• a prebuilt C++ runtime library (libexecutorch.so) you would link against,
• C++-linkable CPU operator kernels (today's kernels are baked into the Python
  extension, not exposed as a standalone library),
• any C++-linkable delegate backend library (for example the CUDA backend).

One nuance to be precise about: the wheel already installs a small header subset
and an executorch-config.cmake. But that config is meant for building custom-op
extensions against the Python module. It defines no libexecutorch.so and no
public runtime, kernel, or delegate targets, so it is not a standalone C++ SDK.

So if you want to write a small C++ program that loads a .pte and runs it on
GPU, your only option is to build ExecuTorch from source: clone the repo, sync a
large set of submodules, install a C++ toolchain, and run a CMake build. That is
a steep first step, it is slow, and it is easy to get wrong. It is a real
barrier for anyone who just wants to try the runtime or ship a product on top of
it.

What we can learn from PyTorch

PyTorch already solved this shape of problem:

• pip install torch gives you prebuilt libtorch shared libraries inside the
  wheel. C++ users link against them, they do not rebuild PyTorch.
• CUDA builds are published to dedicated package indexes (for example a
  cu126-flavored index), and the CUDA runtime components come from declared
  dependency wheels rather than being vendored. ExecuTorch intends to mirror that
  model: accelerator-index selection plus declared CUDA component dependencies,
  rather than inventing a new resolver strategy.

We can mirror this model for ExecuTorch.

Proposal

Bundle prebuilt binaries in the wheel, in composable layers, so a C++ consumer
links only what it needs (installation-wise the artifact is complete; "optional"
here means optional to link, not a separate install):

1. Base runtime, always present in the Linux wheel: the shared ExecuTorch
   runtime plus the C++ headers and a CMake package config, so a C++ app can do
   find_package(executorch) and link executorch::runtime with no source
   build. A fully-delegated model needs the runtime plus the required delegate
   target (for example executorch::cuda_backend), but no CPU-kernel DSO.
2. CPU kernels, optional to link: the kernels DSO ships in the base artifact, but
   a consumer links it only when the model has non-delegated or partially
   delegated ops. It is not a separate package to install.
3. Threadpool, shared: the thread pool is a process-wide singleton, so it lives
   in one small shared library that the kernels and any future delegate link
   dynamically. This guarantees a single pool no matter how many pieces are
   loaded.
4. Delegate backends, each its own shared library that registers itself into the
   runtime when loaded. XNNPACK already ships in the default wheel (baked into the
   Python extension), so the refactor splits it into a shared library that lives
   in the base wheel and is reusable from C++. CUDA is the first backend that is
   genuinely new to C++, and it ships in a separate CUDA wheel. Other backends
   (Core ML, MPS, QNN, and so on) follow the same split.

For the genuinely new C++ delegate, start with CUDA, because it delivers the most
value and has the clearest PyTorch-style precedent.

Refactor first: split the monolith so Python and C++ share binaries

Before adding new prebuilt binaries, we should reshape what we already ship, so
the new C++ pieces and the existing Python pieces reuse one set of components
instead of duplicating them.

Today the Python extension _portable_lib.so is a monolith: it statically links
the core runtime, the data loader, the CPU ops, and every enabled backend
(XNNPACK, Core ML, QNN, and so on) all into one file. That works for import executorch, but it means the runtime and each backend exist only inside that
Python-only binary. If we now add standalone C++ libraries next to it, we get two
copies of everything: one baked into _portable_lib for Python, one standalone
for C++. Two copies of the runtime also means two copies of the backend registry
and two thread pools, which is a correctness problem, not just wasted size.

The registry makes this concrete. Backends register themselves through a static
initializer that runs when their library is loaded (including a late
dlopen), and they register into a single process-wide table that lives in one
place in the core runtime. That "single" only holds if the core runtime is loaded
once. If both _portable_lib and a standalone C++ runtime carry their own copy,
you have two registries and backends can land in the wrong one.

The fix is to invert the dependency:

1. Build each reusable piece as one shared library: the core runtime, the thread
   pool, the CPU kernels, and each backend (for example XNNPACK, then CUDA).
2. Make the C++ SDK (the prebuilt runtime libraries, headers, and CMake config
   that let a standalone C++ app link ExecuTorch without building from source)
   link those shared libraries through CMake targets.
3. Make _portable_lib depend on the same shared libraries (a runtime
   dependency) instead of statically baking its own copies.

The result is one binary per component, reused by both worlds: C++ runtime
developers link them directly, and the Python bindings keep working exactly as
they do today, just on top of the shared components instead of a private
monolith. One core runtime, one registry, one thread pool.

This is a foundation change that touches _portable_lib, the most widely used
artifact in the wheel, so it should land as its own small, reviewable stack
before the delegate wheels pile on top. A reasonable order: core runtime shared
first, then thread pool and kernels, then split backends one at a time (XNNPACK
first, then CUDA), each step keeping import executorch byte-for-byte behaviorally
the same and verifying a single registry and a single thread pool. Only after
that foundation is in place do the CUDA and TensorRT delegate wheels sit cleanly
on top.

Platform scope and portability

This proposal is about the Linux C++ story. It does not try to make prebuilt
binaries work everywhere, because compiled C++ is far less portable than Python.

Apple is out of scope. ExecuTorch already has a separate prebuilt distribution
for Apple platforms through Swift Package Manager, which ships .xcframework
binaries for the Core ML, MPS (Metal), and XNNPACK backends. So macOS and iOS
C++ users are already covered there, and Apple-only backends such as Core ML,
MPS, and MLX are not part of this wheel proposal.

Prebuilt wheels would target Linux x86_64 and Linux aarch64, but aarch64 is
not one binary train: generic Linux aarch64 (SBSA/Thor-class) and Jetson (JetPack
/ L4T) have different CUDA and dependency stacks, so they are distinct release
rows even though both can be in the first milestone. A proposed Phase 0 matrix
(exact versions are release inputs to confirm, one tested combination per row):

Row	Arch	Accelerator	Index / tag	Notes
Linux x86_64 CPU	x86_64	none	default PyPI	the small default install
Linux x86_64 CUDA	x86_64	CUDA <major.minor>	CUDA index, +cu<xxx>	CUDA components from declared CUDA wheel deps
Linux aarch64 SBSA CUDA	aarch64 (SBSA/Thor)	CUDA <major.minor>	CUDA index, +cu<xxx>	server-class ARM, standard manylinux aarch64
Jetson (JetPack/L4T)	aarch64 (Jetson)	JetPack CUDA/TensorRT	JetPack index, +jp<xxx>	CUDA/TensorRT from the pinned JetPack environment

Each row pins its own Python, PyTorch, CUDA, TensorRT, and NumPy versions plus
the platform tag; they are not assumed to share one binary. A compiled .so is
tied to a few things at once, so we would build against a defined baseline, the
same way PyTorch does:

• CPU architecture must match (an x86_64 build does not run on an ARM board).
• glibc floor: build against an old enough glibc (the manylinux standard) so the
  library also loads on newer distros. A distro older than the baseline is not
  supported by the prebuilt wheel.
• C++ ABI must match the consumer's toolchain.

Everything outside that stays a from-source build, exactly as today: Windows,
32-bit systems, distros older than the baseline, and microcontroller or
bare-metal targets (for example Cortex-M), which have no dynamic loader and no
pip at all. The promise is narrow on purpose: one pip install for the common
Linux C++ and GPU case, with the source build still available for everyone else.

What the wheel would contain

First, what the wheel already contains today, so the delta is clear. The current
default wheel ships the Python extension _portable_lib.so with a lot baked into
it: core ATen operators, the optimized/quantized/LLM CPU kernels, and the
XNNPACK backend on all platforms, plus (by platform) Core ML and MPS on
macOS and enabled platform backends on Linux (QNN when its SDK and architecture
are available, OpenVINO per release flags). But all of that is baked in for the
Python path only. None of it is a separate library a C++ program can link.
The wheel does install a header subset and an executorch-config.cmake, but that
config targets custom-op extension builds against the Python module: it exposes
no libexecutorch.so and no runtime/kernel/delegate targets, so a standalone C++
app can link no SDK from today's wheel.

So this proposal does not add XNNPACK or the kernels "for the first time"; they
already exist for Python. It exposes them (and the runtime) as reusable shared
libraries that both the C++ SDK and the Python bindings link, per the refactor
above. After that refactor, the Linux wheel's payload would look like this.

Base runtime wheel (default install):

File	Purpose
executorch/lib/libexecutorch.so	shared C++ runtime (core + extensions)
executorch/lib/libexecutorch_kernels.so	CPU operator kernels (ships in the artifact; optional to link)
executorch/lib/libexecutorch_threadpool.so	shared single thread pool
executorch/lib/libexecutorch_xnnpack_backend.so	XNNPACK CPU delegate (already in the default wheel today, for Python)
executorch/include/...	public C++ headers for the SDK surface
executorch/share/cmake/executorch-config.cmake	find_package support

XNNPACK is in the base wheel because it already ships in the default wheel on all
platforms today (just baked into the Python extension). The refactor makes it a
shared library so both the Python bindings and a C++ app can use the one copy.
Platform-specific backends that already ship today (Core ML and MPS on macOS, and
on Linux QNN when available plus OpenVINO per release flags) would be split out
the same way as follow-ups.

CUDA wheel (a complete, mutually exclusive artifact, not an overlay):

The CUDA wheel is a full executorch wheel that contains the entire base payload
plus the CUDA components. You select it instead of the CPU wheel from a CUDA
index; you do not install two overlapping ExecuTorch distributions on top of each
other.

File	Purpose
executorch/lib/libexecutorch_cuda_backend.so	loadable CUDA delegate
executorch/lib/libextension_cuda.so	one process-wide CUDA stream helper
executorch/backends/cuda/libaoti_cuda_shims.so	CUDA runtime shim

CUDA runtime libraries themselves (for example libcudart) are not vendored into
the ExecuTorch wheel. On standard Linux they come from declared CUDA wheel
dependencies in the package metadata (the way current PyTorch CUDA wheels pull
their CUDA components); on Jetson they come from the pinned JetPack environment.

What each piece contains

So it is clear what the SDK actually includes, not just the file names:

• libexecutorch.so (the core runtime): the program loader and executor plus the
  C++ extensions a standalone app needs to load and run a .pte. Concretely it
  bundles the runtime core, the Module API, the tensor extension, the data
  loaders (buffer, file, mmap, shared-ptr), the FlatTensor .ptd reader, and the
  named-data-map. A fully-delegated model needs this plus the delegate target its
  .pte references (for example executorch::cuda_backend), but no CPU kernels.
• libexecutorch_kernels.so (optional CPU kernels): the CPU operator set,
  optimized ops with a portable fallback, for models that have non-delegated ops.
  This would be a single merged registration, not two libraries stacked together:
  the optimized and portable op lists are deduplicated by op name (optimized wins,
  portable fills the gaps) into one table, so each op registers exactly once.
  Linking the separate portable_ops_lib and optimized_ops_lib instead would
  double-register their shared ops and abort at load, which is why the design ships
  one merged library. Packaging this as a standalone, retained registration DSO
  that both the Python extension and a C++ app link is proposed implementation
  work.
• libexecutorch_threadpool.so (shared thread pool): the single process-wide
  thread pool, linked by the kernels and the XNNPACK backend so they share one
  pool.
• libexecutorch_xnnpack_backend.so (XNNPACK CPU delegate): the fast CPU backend
  most CPU models use. Already in today's default wheel, but baked into the Python
  extension; the refactor makes it a shared library both worlds link.
• Headers: a curated set of public C++ headers, not the whole tree, covering the
  Module, tensor, data-loader, FlatTensor/.ptd, named-data-map, and
  memory-allocator APIs. Curated so we only advertise APIs whose implementation is
  actually shipped in the libraries above.
• CMake package config: find_package(executorch) plus the targets a consumer
  links, such as executorch::runtime, executorch::kernels,
  executorch::threadpool, executorch::xnnpack_backend, and (in the CUDA wheel)
  executorch::cuda_backend.

The CUDA wheel adds the CUDA delegate library, the CUDA stream helper, and the
AOTInductor CUDA shim, and exposes the executorch::cuda_backend target.

Wheel size impact

These are rough design-time estimates to show the order of magnitude, not
measured numbers. Final sizes depend on build flags, symbol tables, RPATHs, and
stripping, and must be produced and approved as a release-gate output before
rollout (see Release gate).

Wheel	Approx. size	Delta
Baseline today (_portable_lib with XNNPACK + kernels baked in, Python only)	~12.5 MB	-
+ shared C++ runtime + headers + cmake	~12.8 MB	+0.3 MB
CUDA wheel (adds the CUDA delegate)	~16.3 MB	+3.8 MB total

Key point: the base runtime adds almost nothing (about 0.3 MB) because the heavy
parts (XNNPACK and the CPU kernels) are already in today's baseline wheel, just
baked into the Python extension. The refactor mostly moves them into shared
libraries rather than adding new weight, so a C++ app can reuse them. The CUDA
delegate is the main genuinely new payload, and it lives in a separate CUDA wheel.

How you would use it (after)

Install:

# CPU / base runtime (default index)
pip install executorch

# CUDA build: pip does not guarantee index priority, so PIN the exact synchronized
# release. The version pin accepts the accelerator-tagged wheel (X.Y.Z+cuXXX) from
# the accelerator index and prevents a newer CPU release on PyPI from winning.
pip install "executorch==X.Y.Z" \
  --index-url https://download.pytorch.org/whl/<accelerator-family> \
  --extra-index-url https://pypi.org/simple

Link a C++ app with CMake, no source build:

find_package(executorch CONFIG REQUIRED)

add_executable(my_app main.cpp)

# Fully-delegated model: runtime + the delegate its .pte uses (e.g. cuda_backend).
target_link_libraries(my_app PRIVATE executorch::runtime)

# Model with CPU ops: add the kernels.
# target_link_libraries(my_app PRIVATE executorch::runtime executorch::kernels)

# CUDA build: add the delegate (from the CUDA wheel).
# target_link_libraries(my_app PRIVATE executorch::runtime executorch::cuda_backend)

Loading a delegate library registers its backend into the runtime, so the same
C++ code runs a CUDA-delegated .pte with no extra wiring.

DevX: before vs after

Step	Today	With this proposal
Get the runtime	clone repo + sync submodules	pip install executorch
Build it	CMake build from source	already prebuilt in the wheel
Toolchain	install a full C++/CUDA toolchain	standard C++ toolchain only, no ExecuTorch/CUDA source build
Time to first run	many minutes, easy to misconfigure	seconds
Link a C++ app	figure out include/lib paths by hand	find_package(executorch)
Use the CUDA backend	build the backend from source	install the CUDA wheel

Ecosystem benefit: Torch-TensorRT

This packaging model does not just help ExecuTorch. Any project that builds a
delegate can publish its own prebuilt backend wheel on top of the base runtime,
using the exact same pattern. Torch-TensorRT is the clearest example.

Torch-TensorRT can export a model where part of the graph runs on a TensorRT
engine and the rest runs on the CUDA backend. Today, to run that in C++, a user
has to build both ExecuTorch and the TensorRT delegate from source and wire them
together by hand. With this proposal, Torch-TensorRT ships its own TensorRT
delegate library (libexecutorch_trt_backend.so) in the
Torch-TensorRT wheel, and depends on a matching ExecuTorch CUDA artifact when the
[executorch] extra is requested. Torch-TensorRT owns and installs that library
in its own package; it does not drop files into the executorch package. Each
project exports its own CMake targets (executorch::runtime,
executorch::kernels, executorch::cuda_backend from ExecuTorch;
torchtrt::executorch_backend from Torch-TensorRT). The TensorRT and CUDA
delegates then load into the same one runtime and run the same .pte together
("coalesced").

Two concrete deliverables this requires on the Torch-TensorRT side: (1) ship a
relocatable Torch-TensorRT CMake package that installs and exports an imported
backend target (today the in-tree target is only an alias and is not installed as
a findable package), and (2) change the wheel build to ship the shared, coreless
delegate DSO and stop packaging a private ExecuTorch core for this path, so the
delegate resolves the runtime from the one libexecutorch.so.

Torch-TensorRT already declares ExecuTorch as an optional [executorch] extra,
but today that extra resolves to the default CPU ExecuTorch wheel, which has no
CUDA delegate. So a user can export a coalesced .pte but cannot pip install
a runtime that runs it on GPU. This proposal is exactly what closes that gap: it
gives the [executorch] extra a CUDA-enabled wheel to resolve to.

What a Torch-TensorRT user would do, end to end:

# Torch-TensorRT pulls in ExecuTorch via its [executorch] extra, which should
# require an exact executorch==X.Y.Z. Pin the Torch-TensorRT release too and point
# pip at the accelerator-specific ExecuTorch index; PyPI is the fallback for
# unrelated deps. Pinning is what prevents a newer CPU release from being chosen.
pip install "torch-tensorrt[executorch]==X.Y.Z" \
  --index-url https://download.pytorch.org/whl/<accelerator-family> \
  --extra-index-url https://pypi.org/simple

# Export: compile with Torch-TensorRT, then save a coalesced .pte. Passing a
# CudaPartitioner after the TensorRT partitioner sends the ops TensorRT does not
# take to ExecuTorch's CUDA backend, instead of leaving them non-delegated.
import torch, torch_tensorrt
from executorch.backends.cuda.cuda_backend import CudaBackend
from executorch.backends.cuda.cuda_partitioner import CudaPartitioner

trt_gm = torch_tensorrt.compile(model, ir="dynamo", arg_inputs=inputs)
torch_tensorrt.save(
    trt_gm, "model.pte", output_format="executorch",
    retrace=False, arg_inputs=inputs,
    partitioners=[
        CudaPartitioner(
            [CudaBackend.generate_method_name_compile_spec("forward")]
        )
    ],
)
# TensorRT claims what it can; CUDA picks up the rest. External CUDA weights are
# written as .ptd file(s) next to the .pte.

// Run in C++: no ExecuTorch/CUDA source build. The coalesced .pte has a TensorRT
// partition and a CUDA partition, and the CUDA partition's weights live in a .ptd
// data file next to the .pte, so both must be provided. The two delegate DSOs are
// pulled into the app at link time (see the CMake below), so they load at startup
// and their static initializers register TensorRTBackend and CudaBackend into the
// one runtime before the program is loaded.
#include <executorch/extension/module/module.h>
using executorch::extension::Module;

int main() {
  // Pass both the program and its external .ptd data map so the CUDA weights
  // resolve. Both backends are already registered via the linked delegate DSOs.
  Module module("model.pte", "model.ptd");
  auto outputs = module.forward(/* inputs */);
  if (!outputs.ok()) return 1;
  // TensorRT engine + CUDA backend ran together; compare against a reference.
}

# Each project ships its own relocatable CMake package.
find_package(executorch CONFIG REQUIRED)
find_package(torch_tensorrt CONFIG REQUIRED)
add_executable(app main.cpp)
# Runtime + CUDA delegate from ExecuTorch, TensorRT delegate from Torch-TensorRT.
# The delegate targets must force retention (scoped --no-as-needed) so the DSOs
# stay in the app's DT_NEEDED and load at startup; static registration then runs.
# Merely naming a target is not enough under default --as-needed. CI verifies both
# DT_NEEDED entries and that both backends are present in the one registry.
target_link_libraries(app PRIVATE
  executorch::runtime executorch::cuda_backend torchtrt::executorch_backend)

Before vs after for a Torch-TensorRT user:

Step	Today	With this proposal
Get ExecuTorch runtime	build from source	comes with the wheel
Get the CUDA backend	build from source	comes with the wheel
Get the TensorRT delegate	build from source	comes with the wheel
Combine TensorRT + CUDA	manual CMake wiring	load two libraries, done
Time to first GPU run	hours of setup	one pip install

The point: once ExecuTorch ships a linkable runtime and an opt-in delegate
pattern, the whole delegate ecosystem (Torch-TensorRT, and others) can deliver a
one-pip install GPU experience without asking users to build anything.

Packaging questions for discussion

These are the open decisions that need consensus before implementation:

1. Index strategy. Mirror PyTorch's index/dependency model: select the artifact
   by index (default index for the CPU build, an accelerator-specific index for
   the CUDA/JetPack build) and pull CUDA components from declared dependency
   wheels. This is about how the artifact is selected, not a claim that the
   default index is permanently CPU-only.
2. Accelerator artifact identity. Which accelerator versions ship at launch, and
   how are they tagged? Proposal: use local version labels like +cu<xxx> for
   CUDA and +jp<xxx> for JetPack, one tested version per row to start, and
   require downstream extras that pull ExecuTorch to pin the compatible release.
3. Platforms. Start Linux-only (x86_64 and aarch64)? macOS already has a
   separate framework-based C++ distribution, and Windows has no C++ SDK story
   yet. Proposal: Linux first.
4. Publishing ownership, rollback, and notification. Who owns each index and
   approves uploads, and what is the response to a bad published wheel? Before
   first publication we need named owners per index, an upload-approval step, a
   yank/removal policy, restoration of the last compatible pins, and a user
   notification path. This is release-infrastructure work separate from the
   packaging code.
5. Delegate rollout order. XNNPACK comes along with the base refactor (it already
   ships today, just needs splitting into a shared library). CUDA is first for the
   genuinely-new C++ delegates, on its own index. Other backends (Core ML, MPS,
   QNN, OpenVINO) split out as demand appears.
6. PyPI size limits. Confirm the base wheel stays comfortably within the project
   size limit; large CUDA wheels live on the separate index anyway.
7. ABI compatibility policy. For the first milestone, do not promise a stable
   cross-release C++ ABI. Use a lockstep contract instead: the runtime, the CUDA
   delegate, and any external backend (such as Torch-TensorRT's) are built and
   tested against the same ExecuTorch release, toolchain ABI, CUDA train, and
   platform. A delegate built for release X loads only with runtime X.
8. Dependency pinning (committed rule, not just open question). Every release row
   pins an exact synchronized public version: Torch-TensorRT's [executorch]
   extra requires executorch==X.Y.Z (which accepts the X.Y.Z+cu<xxx> or
   X.Y.Z+jp<xxx> artifact) rather than an open-ended >=, so a newer,
   ABI-incompatible runtime cannot be selected. Torch-TensorRT currently requires
   executorch>=1.3.1, which is too broad for prebuilt-binary compatibility.
9. Jetson as a first-class Phase 0 row (not deferred). Jetson is a primary target
   for the coalesced GPU use case, so it needs its own release row separate from
   generic Linux aarch64, with one initially tested combination pinned: JetPack
   version, CUDA version, TensorRT version, PyTorch version, Python version, device
   architecture, and the manylinux-or-JetPack packaging assumption. Note two
   concrete blockers to resolve first: (a) a NumPy conflict, since full ExecuTorch
   requires numpy>=2.0.0 while Torch-TensorRT's JetPack requirements pin
   numpy<2.0.0 (no version satisfies both today); and (b) the Torch-TensorRT
   ExecuTorch backend build is currently enabled only for x86_64 and SBSA, so the
   JetPack build target needs explicit enablement and validation.

Release gate

Before any artifact is published, a clean, wheel-only test (no source checkout of
either project) should:

• install the ExecuTorch CUDA/JetPack wheel and the Torch-TensorRT wheel from the
  intended indexes;
• run find_package(executorch CONFIG REQUIRED) and, for the TensorRT path,
  find_package(torch_tensorrt CONFIG REQUIRED);
• compile a C++ app linking executorch::runtime, executorch::cuda_backend, and
  (for TensorRT) torchtrt::executorch_backend;
• inspect the final binary with readelf/nm to confirm exactly one runtime
  registry owner, the expected DT_NEEDED/SONAME/$ORIGIN RUNPATH, and that the
  Python extension and the C++ app resolve to the same registry;
• run the Python-side gates too, since the refactor changes _portable_lib:
  import executorch, a representative .pte execution, the existing Python
  APIs, custom-op compilation through the legacy CMake contract, and default
  (CPU) wheel behavior;
• load a coalesced .pte plus its .ptd, execute the CUDA and TensorRT
  partitions, and compare against a reference;
• run in clean x86_64 and Jetson environments, and block upload if any
  auditwheel, dependency-closure, duplicate-symbol, RPATH, or execution check
  fails.

Every platform row that ships an artifact must pass this gate in its own clean
environment before that artifact is published. Phase 0 ships x86_64 and Jetson;
SBSA/Thor is a follow-up row and is not published until it passes the same gate.
Who owns each index, who approves uploads, and the yank/rollback and
re-pin-to-last-good policy for a bad wheel are open items under Packaging
questions and must be named before first publication.

This is the bar for calling the SDK "done": the packaging direction can be
approved now, but Phase 0 is complete only after this installed-wheel test passes
on both Linux x86_64 and the selected Jetson configuration.

Non-goals

• Vendoring the CUDA runtime (libcudart and friends) into the wheel. That is
  not vendored into the ExecuTorch wheel; they come from declared CUDA wheel
  dependencies (standard Linux) or the pinned JetPack environment (Jetson).
• Bundling every delegate into one giant wheel. Delegates are opt-in libraries.
• Changing the AOT/export flow. This proposal is only about the runtime and
  delegate binaries in the wheel.

Implementation plan (PR stack)

Conceptually the work is a stack of small, independently reviewable PRs, ordered
so each one compiles, keeps import executorch behaving the same, and preserves a
single registry and a single thread pool. It is deliberately foundation-first:
even though the CUDA delegate is the motivation, landing it before the runtime is
shared would ship a second registry and a second thread pool.

Phase A, foundation refactor (ExecuTorch):

1. Ship a linkable libexecutorch.so plus curated headers and a relocatable
   CMake package exposing executorch::runtime. Additive; the Python extension
   is untouched.
2. Make _portable_lib depend on libexecutorch.so instead of statically
   embedding its own runtime copy. This is the key correctness step (one
   registry) and the highest-risk PR, so it lands alone.
3. Split the thread pool into a shared library both worlds link, so there is one
   process-wide pool owner.
4. Ship the merged CPU-kernel registration DSO (executorch::kernels) that both
   the Python extension and a C++ app resolve through.
5. Split XNNPACK into a shared delegate DSO (executorch::xnnpack_backend),
   completing the "one binary per component" base wheel.

Phase B, CUDA delegate (ExecuTorch, separate accelerator index):

6. Ship the coreless CUDA delegate DSOs and executorch::cuda_backend, resolving
   the runtime from libexecutorch.so and taking CUDA components from declared
   dependencies. This depends only on step 1, so it can develop in parallel with
   steps 2 to 5.

Phase C, gates and ecosystem:

7. Add the release-gate CI (ELF/registry checks, coalesced execution, size
   budget, Python-side gates) and the accelerator-index publishing workflow.
8. Torch-TensorRT integration (in the Torch-TensorRT repo): ship a relocatable
   CMake package exporting torchtrt::executorch_backend, ship the shared
   coreless TensorRT delegate, pin executorch==X.Y.Z, and resolve the JetPack
   build-enablement and NumPy version conflicts.

Each Phase A step is small and reversible; the risk concentrates in step 2 (the
_portable_lib dependency inversion) and step 8 (JetPack/NumPy), which is why
they are isolated.

What comes next

If there is agreement on the approach and the packaging questions above,
implementation follows the PR stack above, refactor first. Feedback welcome on
the layering, the refactor ordering, the index strategy, and the initial CUDA
version and platform targets.

[JacobSzwejbka]

What backends would we do this for? Cuda and MLX? Mobile and embedded backends continue as is today?

[shoumikhin]

@JacobSzwejbka start with Cuda only for now. Plus TensorRT coming from torch-tensorrt wheel out of tree. Expand to Xnnpack and other if really needed.

[JakeStevens]

I do think XNNPACK makes sense to lower the bar to people getting started after cuda and trt

[Gasoonjia]

i love the idea my consideration would be the scope; xnnpack + cuda/trt is a good start for me. mlx would be the next step we can reach.

[digantdesai]

Index strategy. Keep the default PyPI wheel CPU-only and publish CUDA wheels
to a dedicated index, mirroring PyTorch? This keeps the common install small.

torch pypi contains libtorch_cuda but only one version IIRC, so we can do the same?

[shoumikhin]

Thanks all, this is really helpful. Folding the feedback into the RFC:

On scope, sounds like we're aligned: CUDA first (+ TensorRT from the torch-tensorrt wheel, out of tree), XNNPACK comes along with the base refactor since it already ships today, and MLX/Core ML stay on the Apple SwiftPM path (out of scope here). I wrote that up in the "Refactor first" and "Platform scope" sections.

@digantdesai yep, we can do exactly what torch does. Default PyPI torch is a single CUDA build and the CPU/other-CUDA variants live on the download.pytorch.org indexes, so "one CUDA version as default + others on a dedicated index" is the model. Two things I made explicit in the RFC after your comment: (1) the accelerator index has to be primary (--index-url), with PyPI only as --extra-index-url fallback, otherwise pip can silently grab the CPU wheel; and (2) CUDA components aren't vendored, they come from declared CUDA wheel deps on standard Linux (same as torch today) or the pinned JetPack env on Jetson. I also added a concrete Phase 0 platform matrix (x86_64, SBSA/Thor, Jetson) and +cu<xxx>/+jp<xxx> version tagging, since Jetson is a different binary train.

Bigger picture the RFC now leads with a refactor-first framing: split the runtime into one shared libexecutorch.so that both _portable_lib and a C++ app depend on (one registry, one thread pool), then layer the CUDA delegate on top. Would love eyes on that section before we lock the design.