POC: OverlayBD image support for on-demand loading and faster cold-start

[BigVan]

Summary

Hello, based on an early version of microsandbox (commit ba02ae334e319faefd8c18dde34bce9822ab3df5), we conducted a POC (Proof of Concept) to validate the use of OverlayBD images as an alternative rootfs backend for microsandbox.

Problem: OCI Image Cold-Start Overhead

When cold-starting a sandbox with a standard OCI image, the entire image must be downloaded and extracted before the sandbox can boot. For multi-layer images, this process is time-consuming and significantly increases startup latency — especially in environments where fast provisioning is critical.

$ time msr python-oci -e 'python -c print("hello world!")'
✓ Fetch image details
✓ Download layers 11 / 11
✓ Extracting layers 11 / 11
hello world!

real    0m20.098s
user    0m7.747s
sys     0m3.187s

What is OverlayBD?

OverlayBD (Overlay Block Device) is a container image format and storage backend developed under the containerd project. It exposes OCI container images as virtual block devices, enabling:

• On-demand image loading — Layer data is fetched from the registry at the block level on demand, eliminating the need for a full image pull. This dramatically reduces cold-start time.

• Writable block device as rootfs — The image is exposed as a virtual block device (/dev/sdX) via TCMU and passed to the microVM through virtio-blk, enabling direct block-level writes without copy-on-write overhead.

• Better I/O performance — virtio-blk provides significantly better small-file read/write performance compared to virtiofs.

• NO COPY-ON-WRITE OVERHEAD — OverlayBD tracks diffs at the sector level. Only the affected sectors are written to the upper layer (data.lsmt), with no file-level copy-up. This is critical for:

  • Online snapshots — only sector-level diffs need to be persisted, making snapshots fast and lightweight.
  • Sandbox forking — a new writable layer on top of the current state is all that's needed; near-zero overhead, no CoW amplification.

 OverlayBD (sector-level diff):
┌──────────────────────┐
│ Write to 1 byte of   │
│ a 100MB file         │
├──────────────────────┤
│ → write 1 sector     │
│   (512B / 4KB)       │
└──────────────────────┘

Performance Comparison

A simple cold-start benchmark over the public internet shows a ~5x improvement with OverlayBD:

Image Type	Cold-Start Time
OCI Image	~20.1s
OverlayBD Image	~3.9s

OverlayBD Image

$ time msr python-overlaybd -e 'python -c print("hello world!")'
✓ Fetch image details
hello world!

real    0m3.949s
user    0m0.193s
sys     0m0.144s

OCI Image

$ time msr python-oci -e 'python -c print("hello world!")'
✓ Fetch image details
✓ Download layers 11 / 11
✓ Extracting layers 11 / 11
hello world!

real    0m20.098s
user    0m7.747s
sys     0m3.187s

Sandboxfile Example

sandboxes:
  python-overlaybd:
    image: 'overlaybd/python:c63f71c375c2_overlaybd'
    memory: 1024
    cpus: 1
    block_device:
      size: 64
      filesystem: ext4
      sparse: true

Next Steps

If the team finds this direction worth pursuing, I'm happy to rebase the work onto the current main branch and submit a PR with a clean implementation.

References

• containerd/overlaybd
• containerd/accelerated-container-image
• Fork branch: BigVan/microsandbox@overlaybd
• Design doc: docs/guides/overlaybd.md

[appcypher]

Thanks for this proposal. I will take a deeper look at it.

I will note however that your use of msr here shows you are using pre-v0.3 version of microsandbox. v0.3 is ground-up reimplementation of microsandbox addressing a lot of the issues we faced.

The upcoming v0.4 will introduce a new disk-based OCI setup but what you introduce here, with on-demand layer pulling, is surely interesting. This is probably an opportunity to start discussion around supporting custom rootfs semantics.

[BigVan]

Overlaybd has been widely used in Alibaba / Azure / Databricks, etc., we are looking forward to further discussions.

[hsiangkao]

Honestly, AFAIK, currently microsandbox already tries to switch erofs+overlayfs since I notice this project this days ago and I never contact them before. So I replied here:

If you really go deep into virtual disk formats, your SOTA should be standard qcow2/vmdk/vhdx (microsandbox libkrun already supports qcow2/vmdk) rather than overlayfs. All those formats support disk snapshots. If you really want to do a fair comparison, those formats should be your baselines to prove your format is best and modern than those.

I really think it's unwise your team repeats those untrue claims all over the internet without the real comparsions to the prior SOTA disk formats (including goharbor/community#165, apple/containerization#116 and e2b-dev/infra#1393): virtual disk layering formats are technologies existing for more than 20 years (I started to use vmware in the 2000s), overlaybd is not the first one, and should not be the last one; but related virtual disk technologies always have their limitation, and that is why people trying to resolve this in the fs instead, especially for containerization use cases, where untrusted remote filesystem data must be imported through shared host kernels.