Problem / motivation
Metaflow can offload step execution to remote compute via @kubernetes and @batch. I'd like to add support for running steps on Tenki Sandbox — disposable Linux microVMs driven by a small SDK (Sandbox.create() → sb.exec("bash", "-c", …)).
For teams already using Tenki, this would let them run Metaflow flows on Tenki microVMs without standing up a Kubernetes cluster or AWS Batch, while keeping the rest of Metaflow (datastore, metadata, cards, @retry, @resources, @catch, resume) unchanged.
Opening this as an issue first per the Core Runtime Contributions section of CONTRIBUTING (the change adds a compute backend and touches metaflow/metaflow_config.py).
Proposed approach
A new @tenki step decorator + backend, modeled 1:1 on the @kubernetes plugin, under metaflow/plugins/tenki/:
TenkiDecorator(StepDecorator) — lifecycle hooks, @resources merge (the @batch compute_resource_attributes idiom), datastore validation.- A click trampoline (
tenki step) + a runner that builds the same bash entrypoint @kubernetes builds (mflog + get_package_commands + bootstrap_commands + save_logs) and runs it via sb.exec(...).
- A thin, lazily-imported wrapper over the
tenki-sandbox SDK — a soft dependency, imported on demand exactly like the Kubernetes/cloud SDKs (not added to install_requires).
The remote-execution contract (the bash string + METAFLOW_* env vars) is backend-agnostic, so most of it is reused verbatim. The Tenki-specific parts:
- Because a microVM has no ambient cloud identity, the backend forwards the matching datastore credentials into the sandbox as env vars (S3/Azure/GS).
- Because
sb.exec is synchronous, the runner runs it in a background thread and tails structured logs from the datastore on the main thread (preserving live log streaming), then maps CommandResult.exit_code / SDK exceptions to Metaflow's retry semantics.
- Sandbox cleanup: in-process teardown (
finally + atexit) + a tenki list / tenki kill CLI (tagged sandboxes) + a server-side max_duration cap as an eventual backstop.
Packaging
I'm proposing this as a core backend, modeled 1:1 on @kubernetes. It's structured so it could instead be repackaged as a metaflow_extensions plugin (only the two registration lines and the config block would move) if you'd rather keep vendor-specific backends out of core — happy to go that route if you prefer.
The one change to a shared core file is a one-line addition of "tenki" to the hardcoded remote-backend list in metaflow/plugins/pypi/conda_decorator.py(the same list that already contains batch, kubernetes, nvidia, …) so that @pypi/@conda treats @tenki like the other remote backends.
Status (already implemented + validated, for context)
I have a working implementation, validated end-to-end against real Tenki microVMs (with MinIO as an S3 datastore): happy path, plain @tenki steps,@retry, concurrent foreach, @timeout, --metadata=service, @catch, resume, @pypi, and orphan cleanup. 41 unit tests; no regressions to existing tests.
The implementation is ready; per the Core Runtime process I'm filing this issue first and will open the PR next.
Problem / motivation
Metaflow can offload step execution to remote compute via
@kubernetesand@batch. I'd like to add support for running steps on Tenki Sandbox — disposable Linux microVMs driven by a small SDK (Sandbox.create()→sb.exec("bash", "-c", …)).For teams already using Tenki, this would let them run Metaflow flows on Tenki microVMs without standing up a Kubernetes cluster or AWS Batch, while keeping the rest of Metaflow (datastore, metadata, cards,
@retry,@resources,@catch,resume) unchanged.Opening this as an issue first per the Core Runtime Contributions section of CONTRIBUTING (the change adds a compute backend and touches
metaflow/metaflow_config.py).Proposed approach
A new
@tenkistep decorator + backend, modeled 1:1 on the@kubernetesplugin, undermetaflow/plugins/tenki/:TenkiDecorator(StepDecorator)— lifecycle hooks,@resourcesmerge (the@batchcompute_resource_attributesidiom), datastore validation.tenki step) + a runner that builds the same bash entrypoint@kubernetesbuilds (mflog +get_package_commands+bootstrap_commands+save_logs) and runs it viasb.exec(...).tenki-sandboxSDK — a soft dependency, imported on demand exactly like the Kubernetes/cloud SDKs (not added toinstall_requires).The remote-execution contract (the bash string +
METAFLOW_*env vars) is backend-agnostic, so most of it is reused verbatim. The Tenki-specific parts:sb.execis synchronous, the runner runs it in a background thread and tails structured logs from the datastore on the main thread (preserving live log streaming), then mapsCommandResult.exit_code/ SDK exceptions to Metaflow's retry semantics.finally+atexit) + atenki list/tenki killCLI (tagged sandboxes) + a server-sidemax_durationcap as an eventual backstop.Packaging
I'm proposing this as a core backend, modeled 1:1 on
@kubernetes. It's structured so it could instead be repackaged as ametaflow_extensionsplugin (only the two registration lines and the config block would move) if you'd rather keep vendor-specific backends out of core — happy to go that route if you prefer.The one change to a shared core file is a one-line addition of
"tenki"to the hardcoded remote-backend list inmetaflow/plugins/pypi/conda_decorator.py(the same list that already containsbatch,kubernetes,nvidia, …) so that@pypi/@condatreats@tenkilike the other remote backends.Status (already implemented + validated, for context)
I have a working implementation, validated end-to-end against real Tenki microVMs (with MinIO as an S3 datastore): happy path, plain
@tenkisteps,@retry, concurrent foreach,@timeout,--metadata=service,@catch,resume,@pypi, and orphan cleanup. 41 unit tests; no regressions to existing tests.The implementation is ready; per the Core Runtime process I'm filing this issue first and will open the PR next.