✨ One pager for KubeVM: A pluggable, provider agnostic APIs for Virtual Machines on Kubernetes - #1893
Conversation
Reference the actual review PR now that it exists, instead of a generic pointer to open PRs against this file.
Point straight at PR vmware-tanzu#1893 without explaining why the file was split out.
* Move KubeVM one-pager content out for PR review The one-pager landed as a single large file, which made it unreviewable as a diff. Replace it with a pointer while the content is reintroduced through a follow-up pull request that shows it as a reviewable addition. * Point one-pager placeholder at PR #1893 Reference the actual review PR now that it exists, instead of a generic pointer to open PRs against this file. * Simplify one-pager placeholder text Point straight at PR #1893 without explaining why the file was split out. * Add please to one-pager placeholder text
Reintroduce the full one-pager content on top of the placeholder so this PR shows the proposal as a reviewable diff instead of a single opaque file.
f09156b to
26ba100
Compare
github-actions
Bot
added
size/L
|
|
||
| ## Business Problem | ||
|
|
||
| Kubernetes has become the default control plane for modern infrastructure, yet the ecosystem still lacks a cross-platform, VM-centric API: a single declarative surface through which any hypervisor or cloud can expose both the full lifecycle of a virtual machine and the hardware capabilities that demanding workloads depend on. |
There was a problem hiding this comment.
How do we prevent this from being a least common denominator API?
Also what are the thoughts on leaky abstractions because folks will ALWAYS want to configure/tweak things below especially calling out security or performance
There was a problem hiding this comment.
How do we prevent this from being a least common denominator API?
As with any new standard, I expect some challenges with this early on. But, as we start to show value by bringing in abstractions on top of the virtual machine layer, I expect more and more providers to converge into integrating into this API.
Taking the same example of VirtualMachineDeployments: If the KubeVM orchestration layer handles creation, rollout and deletion of the VMs of the Deployment, every provider gains this feature without doing any work at all.
Also what are the thoughts on leaky abstractions because folks will ALWAYS want to configure/tweak things below especially calling out security or performance
This is a fair point and this will always be a discussion. For a provider that does a certain feature in a completely bespoke manner, they can continue to handle that via the provider APIs. However, in doing so, they would lose the benefits offered by the KubeVM API.
There was a problem hiding this comment.
I hear you and this is fine for now.
BUT i do want you to think from the point of view of the people who will use this, formulating this as a Business Problem is thinking from the vendor perspective.
How is this going to help someone? Can they switch between multiple kubernetes clusters that run on different infra providers? how easy would it be for them to switch between them? How do they specify say the underlying image? Will it help manage capacity by bursting into another provider when their primary cluster is full?
Do you think that is worth a section as well above?
There was a problem hiding this comment.
Hi @dims !
I went ahead and modified the doc since the value proposition was not immediately clear. Please take a look and let me know if the doc expresses that more clearly now.
To summarize:
There's a user story section now (following the KEP model) that talks about how this will allow customers to port their virtual machines from one infra provider to the other with minimal changes. The cross provider scheduling at-burst is also a very real use-case that you pointed out.
Regarding the "how easy would it be for them to switch":
This depends on the virtual machine. If a virtual machine depends on a lot of platform specific features not supported by the KubeVM generic layer yet, then the porting to another provider will not be smooth. But as we work with other providers to have common aspects ported to the generic layer, the cross-provider switch will get a lot easier.
|
|
||
| Kubernetes has become the default control plane for modern infrastructure, yet the ecosystem still lacks a cross-platform, VM-centric API: a single declarative surface through which any hypervisor or cloud can expose both the full lifecycle of a virtual machine and the hardware capabilities that demanding workloads depend on. | ||
| This gap is becoming urgent because a new class of workload is arriving faster than the tooling to run it. | ||
| Agentic workloads — long-running processes that execute model-generated code and orchestrate tools — are increasingly deployed inside virtual machines, both for the strong isolation a VM provides around untrusted code and for direct access to the hardware accelerators, such as GPUs, SR-IOV network functions, and passthrough devices, that hypervisors already virtualize well. |
There was a problem hiding this comment.
Is the effort here then ... consistent / exact / replicatable environment across providers?
There was a problem hiding this comment.
Precisely. The goal is to get to a model of what I am calling "thick core with thin provider". Which basically means most of the fields land up in the generic KubeVM API layer with very minor platform specific things landing in the provider resources.
The value proposition is a portable resource. A much stronger value proposition is abstractions built on top of these resource templates. E.g., a VirtualMachineReplicaSet resource that knows how to orchestrate VM replicas from templates.
| Kata Containers provides VM-strength isolation for individual workloads by wrapping a Pod in a lightweight micro-VM — well suited to isolating untrusted code at the granularity of a container. | ||
| KubeVirt takes a different approach, converging the virtual machine into the container model by running a QEMU/KVM process inside a Pod, which is an excellent fit when Kubernetes is the sole infrastructure layer and rich, device-level VM modeling on Kubernetes nodes is the goal. | ||
| Both are strong at their design point. | ||
| What neither sets out to be is a portable, Kubernetes-native front door to a full-blown, hypervisor-native estate — an existing vSphere deployment or a public-cloud VM service — that exposes that platform's own lifecycle and hardware capabilities (GPUs, SR-IOV, passthrough) through one vendor-neutral API. |
There was a problem hiding this comment.
Here are we saying we don't care where we are running? (like virtual kubelet creating things outside of the nodes in the k8s cluster itself?)
There was a problem hiding this comment.
Exactly. It is up to the provider to figure out where and how the virtual machines are instantiated. For a provider like Kubevirt, it will be on one of the nodes. For a provider like vSphere / EC2 / GCP, it can be a "true" VM directly on the hypervisor.
| KubeVirt takes a different approach, converging the virtual machine into the container model by running a QEMU/KVM process inside a Pod, which is an excellent fit when Kubernetes is the sole infrastructure layer and rich, device-level VM modeling on Kubernetes nodes is the goal. | ||
| Both are strong at their design point. | ||
| What neither sets out to be is a portable, Kubernetes-native front door to a full-blown, hypervisor-native estate — an existing vSphere deployment or a public-cloud VM service — that exposes that platform's own lifecycle and hardware capabilities (GPUs, SR-IOV, passthrough) through one vendor-neutral API. | ||
| That is the gap KubeVM fills, and it is complementary to both. |
There was a problem hiding this comment.
What's the stance on making kubernetes resources being made available to things in the VM? (secrets may be handy? env variables? how about CSI volumes?)
Are we limited to what is available in the underlying provider API?
There was a problem hiding this comment.
Good point. Thank you for calling this out. KubeVM will provide the abstraction that lets providers accomplish these tasks. For example: a spec.volume reference with a SecretRef can be consumed by the provider to mount the secret as a volume (however they choose to do so).
Are we limited to what is available in the underlying provider API?
Yes, to some extent we will be limited by what a provider (and the underlying platform / hypervisor) can support.
There are opportunities to standardize the API for common lifecycle operations across all providers dealing with VMs. Bootstrapping (cloud-init / Sysprep) comes to mind. We will work with providers on this.
There was a problem hiding this comment.
I really think that you all have to have a heart-to-heart with the kubevirt folks. give them a demo, walk them through your solution/api et al. figure out overlap areas, where you are different, talk about future plans etc.
| That is the gap KubeVM fills, and it is complementary to both. | ||
|
|
||
| This document proposes **KubeVM**, a generic and vendor-neutral `VirtualMachine` API served under the `kube-vm.io` group, together with a provider model that lets hypervisors and cloud VM services expose their machines — and, critically, their accelerators — through one portable, Kubernetes-native interface. | ||
| KubeVM is intended to complement KubeVirt, not to replace it: it addresses the hypervisor-native design point that the VM-as-Pod model leaves unaddressed. |
There was a problem hiding this comment.
Do the requested VM(s) run inside existing kubernetes nodes? do we have a stance on that?
There was a problem hiding this comment.
The proposal is agnostic to that. If KubeVM is managing KubeVirt resources, then it will leave the Kubernetes layer orchestrate the scheduling of the VM. If it is another provider such as vSphere, or AWS, the underlying hypervisor layer may want to orchestrate the provisioning and management of VMs.
There was a problem hiding this comment.
I reached out to Kubevirt folks to share the proposal in the #virtualization channel in the Kubernetes Slack. I will keep you posted.
Adopt a light KEP-style spine so the document is navigable to reviewers coming from k/enhancements, and answer review feedback that the framing was written from the vendor's side rather than the user's. Structure: - Business Problem becomes Summary + Motivation, with Goals reduced to terse outcomes. - The "As a ..." statements move to Proposal / User Stories, and gain stories for naming an image, moving a workload to a cluster on different infrastructure, bursting into a second provider, and governing what a namespace may ask for. - Big Picture is demoted under Proposal; Architecture Areas becomes Design Details. - New Risks and Mitigations section addresses the least-common- denominator and leaky-abstraction critiques, the KubeVirt overlap, contract drift, and the bias of designing around one provider. - New Alternatives section covers extending KubeVirt, reusing the Cluster API Machine, per-platform CRDs, and single-vendor facades. Beyond a single machine: - Sets now lead with the /scale subresource, which makes the existing HPA and kubectl scale work unchanged rather than implying KubeVM would ship a VM autoscaler. - VirtualMachineService leads with the point that the generic layer implements no load balancing at all, and inherits every existing LB controller by emitting core objects. - Add portable quota, recording the reserved-versus-used lesson StoragePolicyQuota had to learn on Supervisor. - Add portable sizing and configuration policy, enforced at admission before a provider object exists. - Add VirtualMachineHealthCheck, warm pools, and scheduled lifecycle / TTL. - New "Who implements these" states that the generic layer owns the capabilities it implements and does not offer providers a parallel path, with capability advertisement held back until asked for. - New "Capacity-aware placement across providers" covers the bursting question, grounded in the WaitingForCapacity and Preempted reasons the contract already defines, and states the limits: it places new machines only, network identity does not follow, and it presumes the catalog resolves at both targets.
Kubevm/onepager abstractions
github-actions
Bot
added
size/XL
2 participants
What does this PR do, and why is it needed?
KubeVM is a generic set of Kubernetes-native, hypervisor agnostic APIs to deploy and manage the lifecycle of virtual machines from a Kubernetes cluster. This project provides a consistent interface allowing infrastructure providers to expose core platform capabilities, including scheduling, performance, storage, migration and more. The APIs also allow the platform team to define guardrails and policies around the compute, storage, networking, placement of virtual machines.
Are there any special notes for your reviewer:
N / A