-
Notifications
You must be signed in to change notification settings - Fork 692
Deploy fails on image prune when docker rmi returns non-zero #1799
Description
Summary
Deploy reports failure at the image pruning step even though the container is healthy and serving traffic. The root cause is that Kamal's pruning logic is internally contradictory: it intentionally keeps stopped containers but then tries to remove the images they reference.
Steps to reproduce
- Deploy an app with Kamal (2.10.1) multiple times
- Stopped containers accumulate (Kamal keeps the 5 most recent via
tail -n +6) - The image prune step tries to remove images still referenced by those kept containers
docker rmifails with exit code 1, and the deploy is marked as failed
INFO Running docker ps -q -a --filter label=service=myapp ... | tail -n +6 | while read container_id; do docker rm $container_id; done
INFO Finished in 0.440 seconds with exit status 0 (successful).
INFO Running docker image ls --filter label=service=myapp --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' ...)" | while read image tag; do docker rmi $tag; done
ERROR (SSHKit::Command::Failed): Exception while executing on host: docker exit status: 1
The container is running and healthy at this point — the deploy succeeded, but Kamal reports failure.
Root cause
Two issues in lib/kamal/commands/prune.rb:
-
Contradictory pruning logic: The container prune keeps the 5 most recent stopped containers (
tail -n +6), but the image prune then tries to remove images those containers reference. Thegrep -vfilter is supposed to exclude referenced images but doesn't catch all of them. -
No error tolerance: The
while read image tag; do docker rmi $tag; doneloop has no error handling. If any singledocker rmifails, the entire deploy is marked as failed — even though the app is healthy and serving traffic.
Suggested fix
Either:
- Fix the grep filter so it properly excludes images referenced by the kept stopped containers
- Or remove stopped containers before pruning images so there are no references blocking removal
- And add
|| trueto thedocker rmiloop so image pruning is best-effort and never fails the deploy
Environment
- Kamal 2.10.1
- Docker 28.x
- Single-server deploy with local registry