Skip to content

Ensure only RAW images in cache(imgconv/raw) while keeping the original image#4637

Open
unsuman wants to merge 1 commit intolima-vm:masterfrom
unsuman:feature/disk-cache
Open

Ensure only RAW images in cache(imgconv/raw) while keeping the original image#4637
unsuman wants to merge 1 commit intolima-vm:masterfrom
unsuman:feature/disk-cache

Conversation

@unsuman
Copy link
Copy Markdown
Member

@unsuman unsuman commented Feb 26, 2026

Now when a fresh image is downloaded and if it's a non-ISO image, and the image type not supported by the driver it is then converted to RAW in the cache(imgconv/raw) and a raw.digest file is created. It also works for existing images in the cache that are not supported by the driver

Fixes #4056

@unsuman
Copy link
Copy Markdown
Member Author

unsuman commented Feb 26, 2026

Are RAW images not supported on Windows?

@AkihiroSuda
Copy link
Copy Markdown
Member

Are RAW images not supported on Windows?

https://learn.microsoft.com/en-us/windows/win32/fileio/sparse-file-operations says:

It is up to the application to maintain sparseness by writing zeros with FSCTL_SET_ZERO_DATA.

Maybe it is not implemented in qemu-img.exe ?

Comment thread pkg/downloader/downloader.go
Comment thread pkg/downloader/downloader.go Outdated
@unsuman
Copy link
Copy Markdown
Member Author

unsuman commented Feb 26, 2026

Maybe it is not implemented in qemu-img.exe ?

Yes seems so, as you pointed out

Is a .img file qcow2 or raw? I checked it on MacOS it shows raw, does it behave differently on Windows?

Comment thread pkg/driverutil/disk.go Outdated
@AkihiroSuda
Copy link
Copy Markdown
Member

Is a .img file qcow2 or raw? I checked it on MacOS it shows raw, does it behave differently on Windows?

Can't predict the format from the extension

@AkihiroSuda
Copy link
Copy Markdown
Member

https://github.com/lima-vm/lima/actions/runs/22440179260/job/64980256758?pr=4637

qemu-img: SetEndOfFile error: There is not enough space on the disk.

Does --preallocation flag work for this on Windows?
https://qemu-project.gitlab.io/qemu/tools/qemu-img.html

@unsuman
Copy link
Copy Markdown
Member Author

unsuman commented Feb 26, 2026

https://github.com/lima-vm/lima/actions/runs/22440179260/job/64980256758?pr=4637

qemu-img: SetEndOfFile error: There is not enough space on the disk.

Does --preallocation flag work for this on Windows? https://qemu-project.gitlab.io/qemu/tools/qemu-img.html

Gemini tells me:

The --preallocation=off flag is actually the default behavior when extending a file's size. On Linux and macOS, this works perfectly because the underlying system call (ftruncate) naturally creates a "hole" (a sparse file) without consuming physical blocks.

However, on Windows, QEMU translates this resize request into the Win32 API call SetEndOfFile.
Windows NTFS is extremely strict. If a file has not been explicitly branded with the FSCTL_SET_SPARSE attribute before SetEndOfFile is called, Windows assumes you want real, contiguous physical space.

Comment thread pkg/driverutil/disk.go
Comment thread pkg/downloader/downloader.go Outdated
@unsuman unsuman force-pushed the feature/disk-cache branch 3 times, most recently from b3d02da to f0bfc72 Compare March 10, 2026 22:37
@unsuman unsuman added this to the v2.1.0 milestone Mar 10, 2026
@unsuman unsuman force-pushed the feature/disk-cache branch from f0bfc72 to 07cac94 Compare March 10, 2026 23:26
@unsuman
Copy link
Copy Markdown
Member Author

unsuman commented Mar 10, 2026

Current flow:

- Initial state. No cache here.
- vz driver requires https://example.com/foo.qcow2
- Lima downloads the qcow2 image
- Lima converts the qcow2 to the raw cache(imgconv/raw)
- Lima returns the cached raw image
- qemu driver requires the same URL
- +++ Lima returns the qcow2 image because the driver supports it

- User prunes the cache
- qemu driver requires the same URL again
- Lima downloads the qcow2 image
- Lima returns the cached qcow2 image, without any conversion
- vz driver requires the same url
- Lima converts the .qcow2 image to raw and caches it for future use

Only left with adding tests to pkg/downloader/downloader_test.go, I'll complete it before our meeting #4678

Also need to update internals.md

@unsuman unsuman changed the title Ensure only RAW images in cache Ensure only RAW images in cache(imgconv/raw) while keeping the original image Mar 10, 2026
@unsuman unsuman force-pushed the feature/disk-cache branch 4 times, most recently from a27e53b to e03a4c2 Compare March 11, 2026 18:15
Comment thread pkg/fileutils/download.go Outdated
Comment thread pkg/fileutils/download.go Outdated
Comment thread pkg/driver/driver.go
Comment thread pkg/driver/vz/vz_driver_darwin.go
@AkihiroSuda AkihiroSuda removed this from the v2.1.0 milestone Mar 12, 2026
@jandubois

This comment was marked as duplicate.

jandubois added a commit to jandubois/lima that referenced this pull request Mar 19, 2026
Move 2 lima reviews (shell-sync functionality review, PR lima-vm#4637) and
their HTML versions into the per-repo reviews branch. Includes a
placeholder index.html adapted from the old multi-repo index.

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
jandubois added a commit to jandubois/lima that referenced this pull request Mar 19, 2026
Add header table to shell-sync review (old format). Add PR title to

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
lima-vm#4637. Regenerate index.html with generate-index tool.
jandubois added a commit to jandubois/lima that referenced this pull request Mar 19, 2026
Add header table to shell-sync review (old format). Add PR title to

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
lima-vm#4637. Regenerate index.html with generate-index tool.
@AkihiroSuda AkihiroSuda added this to the v2.2.0 milestone Apr 1, 2026
@jandubois
Copy link
Copy Markdown
Member

I improved my review agent to let subagents run more multiple passes as long as they keep finding significant additional issues.

More comprehensive report: Deep Review: 20260327-223819-pr-4637

Signed-off-by: Ansuman Sahoo <anshumansahoo500@gmail.com>
@unsuman unsuman force-pushed the feature/disk-cache branch from e03a4c2 to 5a664e9 Compare April 15, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache should convert to RAW when needed

3 participants