-
Notifications
You must be signed in to change notification settings - Fork 693
Remote BuildKit volume grows unbounded, no built-in pruning #1794
Description
Problem
When using builder.remote, Kamal creates a BuildKit container on the target server and reuses it across deploys for layer caching. However, the BuildKit volume grows with every deploy and is never pruned by Kamal.
On a small VPS (e.g., Hetzner CPX22 — 80 GB disk), the BuildKit state volume can grow to 40–50 GB over a few weeks of regular deploys, consuming most of the available disk space.
Critically, the existing kamal prune command and docker system prune do not reclaim this space — the data lives inside a named Docker volume (buildx_buildkit_*_state), not in the build cache layer that docker buildx prune targets.
Current workaround
A weekly cron job to cap the cache:
0 5 * * 0 docker buildx prune --keep-storage=5GB -f >> /var/log/docker-cleanup.log 2>&1If the volume has already grown too large, the only fix is to remove the BuildKit container and volume entirely (losing all cache):
docker stop $(docker ps -qf name=buildkit) && docker rm $(docker ps -aqf name=buildkit) && docker volume rm $(docker volume ls -q --filter name=buildkit)Suggestion
A few options that could address this at the Kamal level:
- Add a
builder.cache.max_storageoption indeploy.yml— Kamal would rundocker buildx prune --keep-storage=<value> -fafter each deploy or as part ofkamal prune - Include BuildKit pruning in
kamal prune— the existing prune command handles old images and containers but not BuildKit - Document the issue — at minimum, mention that remote BuildKit volumes grow unbounded and suggest a cron-based workaround
Environment
- Kamal 2
builder.remote: ssh://root@<ip>(remote build on same VPS)- Hetzner CPX22 (2 vCPU, 4 GB RAM, 80 GB disk)
- Observed BuildKit volume size after ~2 weeks of deploys: 48 GB