Skip to content

Remote BuildKit volume grows unbounded, no built-in pruning #1794

@yshmarov

Description

@yshmarov

Problem

When using builder.remote, Kamal creates a BuildKit container on the target server and reuses it across deploys for layer caching. However, the BuildKit volume grows with every deploy and is never pruned by Kamal.

On a small VPS (e.g., Hetzner CPX22 — 80 GB disk), the BuildKit state volume can grow to 40–50 GB over a few weeks of regular deploys, consuming most of the available disk space.

Critically, the existing kamal prune command and docker system prune do not reclaim this space — the data lives inside a named Docker volume (buildx_buildkit_*_state), not in the build cache layer that docker buildx prune targets.

Current workaround

A weekly cron job to cap the cache:

0 5 * * 0 docker buildx prune --keep-storage=5GB -f >> /var/log/docker-cleanup.log 2>&1

If the volume has already grown too large, the only fix is to remove the BuildKit container and volume entirely (losing all cache):

docker stop $(docker ps -qf name=buildkit) && docker rm $(docker ps -aqf name=buildkit) && docker volume rm $(docker volume ls -q --filter name=buildkit)

Suggestion

A few options that could address this at the Kamal level:

  1. Add a builder.cache.max_storage option in deploy.yml — Kamal would run docker buildx prune --keep-storage=<value> -f after each deploy or as part of kamal prune
  2. Include BuildKit pruning in kamal prune — the existing prune command handles old images and containers but not BuildKit
  3. Document the issue — at minimum, mention that remote BuildKit volumes grow unbounded and suggest a cron-based workaround

Environment

  • Kamal 2
  • builder.remote: ssh://root@<ip> (remote build on same VPS)
  • Hetzner CPX22 (2 vCPU, 4 GB RAM, 80 GB disk)
  • Observed BuildKit volume size after ~2 weeks of deploys: 48 GB

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions