Skip to content

Deploy fails on image prune when docker rmi returns non-zero #1799

@6temes

Description

@6temes

Summary

Deploy reports failure at the image pruning step even though the container is healthy and serving traffic. The root cause is that Kamal's pruning logic is internally contradictory: it intentionally keeps stopped containers but then tries to remove the images they reference.

Steps to reproduce

  1. Deploy an app with Kamal (2.10.1) multiple times
  2. Stopped containers accumulate (Kamal keeps the 5 most recent via tail -n +6)
  3. The image prune step tries to remove images still referenced by those kept containers
  4. docker rmi fails with exit code 1, and the deploy is marked as failed
INFO Running docker ps -q -a --filter label=service=myapp ... | tail -n +6 | while read container_id; do docker rm $container_id; done
INFO Finished in 0.440 seconds with exit status 0 (successful).

INFO Running docker image ls --filter label=service=myapp --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' ...)" | while read image tag; do docker rmi $tag; done
ERROR (SSHKit::Command::Failed): Exception while executing on host: docker exit status: 1

The container is running and healthy at this point — the deploy succeeded, but Kamal reports failure.

Root cause

Two issues in lib/kamal/commands/prune.rb:

  1. Contradictory pruning logic: The container prune keeps the 5 most recent stopped containers (tail -n +6), but the image prune then tries to remove images those containers reference. The grep -v filter is supposed to exclude referenced images but doesn't catch all of them.

  2. No error tolerance: The while read image tag; do docker rmi $tag; done loop has no error handling. If any single docker rmi fails, the entire deploy is marked as failed — even though the app is healthy and serving traffic.

Suggested fix

Either:

  • Fix the grep filter so it properly excludes images referenced by the kept stopped containers
  • Or remove stopped containers before pruning images so there are no references blocking removal
  • And add || true to the docker rmi loop so image pruning is best-effort and never fails the deploy

Environment

  • Kamal 2.10.1
  • Docker 28.x
  • Single-server deploy with local registry

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions