Skip to content

[Provisioner] Docker image pull is completely silent during launch — multi-GB images look hung for minutes #10141

Description

@ahluwalij

Problem

When launching a cluster with a docker image (image_id: docker:...), every docker setup command is run with stream_logs=False (sky/provision/docker_utils.py, DockerInitializer._run). That is the right default for probe/identity commands (whoami, inspect, daemon-wait retries) whose outputs are parsed, but it also silences docker pull — the one setup step whose duration scales with image size.

For a multi-GB image (e.g. anyscale/ray-llm, vLLM images, CUDA training images), the provision log shows nothing between:

Instance is up.

and

Docker container is up.

for many minutes. Anyone tailing the launch — sky launch clients, the dashboard request log, or an API server's /api/stream — sees a launch that looks hung, with no way to distinguish "pulling 20 GB of layers" from "wedged". The pull's non-TTY output is bounded, line-oriented layer progress, so streaming it is cheap and readable.

Reproduction

resources:
  image_id: docker:vllm/vllm-openai:latest

sky launch on any cloud with --cluster + watch the logs: silence from "Instance is up." until the container starts.

Proposal

Add a stream_logs passthrough to DockerInitializer._run (default False, so all existing call sites keep today's behavior) and enable it only for the two docker pull call sites in initialize(). Output is teed, so it still lands in the per-instance provision log and is still returned to callers for parsing.

Happy to contribute the PR (ready to submit).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions