Ansible Practice Lab: Master 1000+ Docker Containers at Scale

A complete, production-ready guide to building an Ansible automation lab with 1 master node and 1000 target containers on a single GCP VM. This repository documents the entire setup process, from infrastructure provisioning to advanced playbook execution.

---

🎯 Overview

This lab environment enables DevOps engineers and Ansible practitioners to:

• Practice Ansible automation on a massive scale (1000+ managed nodes)
• Test configuration management workflows without cloud costs
• Learn inventory management, playbooks, and best practices
• Prepare for production Ansible deployments

Lab Capacity: 600-1000 containers on a single e2-standard-16 VM (16 vCPUs, 64 GB RAM, 500 GB disk)

---

📋 Prerequisites

Infrastructure

• VM: Google Cloud Platform e2-standard-16 instance (or equivalent)
  • 16 vCPUs
  • 64 GB RAM
  • 500 GB SSD
• OS: Ubuntu 22.04 LTS or later
• Network: Public IP for SSH access

Local Tools

• SSH client
• Docker CLI (installed on VM)
• Text editor (vim, nano, or similar)

---

🚀 Quick Start

1. Provision the Host VM

# Update system packages
apt update -y && apt upgrade -y

# Install Docker and utilities
apt install -y vim docker.io curl git

2. Start the Ansible Master Container

# Create and start master container
docker run -itd --name master ubuntu:latest bash

# Install Ansible inside master
docker exec -it master bash -c '
  apt update -y && apt upgrade -y
  apt install -y python-is-python3 vim iputils-ping openssh-server openssh-client sshpass
  apt install -y software-properties-common
  add-apt-repository --yes --update ppa:ansible/ansible
  apt install -y ansible
  ansible --version
'

3. Build Target Container Image

Save the following as Dockerfile.target:

FROM ubuntu:latest

ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Kolkata

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    tzdata openssh-server openssh-client python3 python-is-python3 sudo vim iputils-ping && \
    ln -fs /usr/share/zoneinfo/Asia/Kolkata /etc/localtime && \
    dpkg-reconfigure --frontend noninteractive tzdata && \
    rm -rf /var/lib/apt/lists/*

RUN mkdir /var/run/sshd && \
    echo 'root:admin' | chpasswd && \
    sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \
    sed -i 's/#PasswordAuthentication yes/PasswordAuthentication yes/' /etc/ssh/sshd_config

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Build the image:

docker build -f Dockerfile.target -t target-img .

4. Deploy 1000 Target Containers

Save the following as deploy_containers.sh:

#!/bin/bash
echo "Deploying 1000 target containers..."
for i in $(seq 1 1000); do
  docker run -d --name target-$i target-img
  [ $((i % 100)) -eq 0 ] && echo "Deployed $i containers..."
done
echo "Deployment complete!"

Run the deployment:

chmod +x deploy_containers.sh
./deploy_containers.sh

# Verify
docker ps | grep target | wc -l

5. Generate Ansible Inventory

Save the following as generate_inventory.sh:

#!/bin/bash

INVENTORY_FILE="hosts"
IP_FILE="ips.txt"

echo "[targets]" > "$INVENTORY_FILE"
> "$IP_FILE"

for i in $(seq 1 1000); do
  name="target-$i"
  ip=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$name" 2>/dev/null)
  
  [ -z "$ip" ] && continue

  echo "${name} ansible_host=${ip} ansible_user=root" >> "$INVENTORY_FILE"
  echo "${ip}" >> "$IP_FILE"
done

echo "Generated inventory with $(grep -c '.' $INVENTORY_FILE) hosts"

Run and copy to master:

chmod +x generate_inventory.sh
./generate_inventory.sh

docker cp hosts master:/etc/ansible/hosts
docker cp ips.txt master:/etc/ansible/ips.txt

6. Setup SSH Key-Based Authentication

Inside the master container:

docker exec -it master bash
cd /etc/ansible
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa

Create distribute_keys.sh inside master:

#!/bin/bash
export SSHPASS='admin'

echo "Distributing SSH keys to all targets..."
count=0
for ip in $(grep -v '^#' /etc/ansible/ips.txt); do
    sshpass -e ssh-copy-id -o StrictHostKeyChecking=no root@$ip 2>/dev/null
    ((count++))
    [ $((count % 100)) -eq 0 ] && echo "Keys distributed to $count hosts..."
done
echo "SSH key distribution complete!"

Make executable and run:

chmod +x /etc/ansible/distribute_keys.sh
/etc/ansible/distribute_keys.sh

7. Verify Ansible Connectivity

ansible all -i hosts -m ping

Expected output: All targets return "ping": "pong" ✓

---

📚 Example Playbooks

Playbook 1: System Information Gathering

File: gather_info.yml

---
- name: Gather System Information
  hosts: targets
  gather_facts: yes

  tasks:
    - name: Get Hostname and IP
      debug:
        msg: "{{ ansible_hostname }} | {{ ansible_all_ipv4_addresses | first }}"

    - name: Get OS Information
      debug:
        msg: "{{ ansible_distribution }} {{ ansible_distribution_version }}"

Run:

ansible-playbook -i hosts gather_info.yml

Playbook 2: Package Management

File: patch_systems.yml

---
- name: Patch All Systems
  hosts: targets
  become: yes

  tasks:
    - name: Update package cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Upgrade all packages
      apt:
        upgrade: dist
        autoremove: yes
        autoclean: yes

    - name: Check if reboot required
      stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot if necessary
      reboot:
        msg: "Rebooting due to kernel updates"
        connect_timeout: 5
        reboot_timeout: 300
        post_reboot_delay: 30
      when: reboot_required.stat.exists

Run with limited parallelism:

ansible-playbook -i hosts patch_systems.yml --forks 50

Playbook 3: Install Custom Software

File: install_packages.yml

---
- name: Install Development Tools
  hosts: targets
  become: yes

  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Install packages
      apt:
        name:
          - curl
          - wget
          - git
          - vim
          - htop
          - net-tools
        state: present

---

⚙️ Performance Tuning

Optimize Parallelism

# Increase simultaneous connections (default: 5)
ansible-playbook -i hosts playbook.yml --forks 100

Adjust System Limits

On the host VM:

# Increase file descriptors
ulimit -n 65535

# Make permanent (add to /etc/security/limits.conf)
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf

Monitor Docker Resource Usage

# Check memory and CPU usage
docker stats --no-stream

# List top memory-consuming containers
docker stats --no-stream | sort -k4 -h | tail -20

---

🔍 Troubleshooting

Issue: Containers Won't Start

# Check container logs
docker logs target-1

# Inspect container state
docker inspect target-1 | grep -A 5 "State"

Issue: SSH Connection Refused

# Test SSH connectivity from master
docker exec master ssh -v root@<target-ip>

# Verify keys were copied
docker exec master ssh -i /root/.ssh/id_rsa root@<target-ip> "echo OK"

Issue: Ansible Ping Fails

# Check if Python is installed on targets
docker exec target-1 python --version

# Test raw SSH command
ansible all -i hosts -m raw -a "whoami"

Issue: Out of Memory

# Reduce number of containers or increase forks limit
ansible-playbook -i hosts playbook.yml --forks 20

# Check memory usage
free -h
docker stats --no-stream | head -20

---

📊 Capacity Planning

Component	Capacity	Bottleneck
Memory	~60 GB usable	Idle: 1200 containers, Active: 600 containers
CPU	16 vCPUs	Concurrent playbook execution
Disk	~30 GB for 1000 containers	Minimal (Docker uses layered filesystem)
Network	VM-dependent	SSH key distribution, large file transfers

---

🛠️ Advanced Topics

Dynamic Inventory

Use Docker API to dynamically discover containers:

# docker_inventory.yml
plugin: community.docker.docker_containers
docker_host: unix://var/run/docker.sock
compose:
  ansible_host: NetworkSettings.Networks.bridge.IPAddress
  ansible_user: root
groups:
  targets: "'target-' in Name"

Role-Based Playbooks

Organize playbooks using Ansible roles for complex deployments:

roles/
  ├── base/
  │   └── tasks/main.yml
  ├── security/
  │   └── tasks/main.yml
  └── monitoring/
      └── tasks/main.yml

Vault for Secrets Management

# Create encrypted variables
ansible-vault create secrets.yml

# Run playbook with vault
ansible-playbook -i hosts playbook.yml --ask-vault-pass

---

📖 Best Practices

1. Use Dynamic Inventory: Avoid manual inventory updates for large deployments
2. Limit Parallelism: Start with --forks 20-50 to avoid overwhelming the system
3. Test on Small Groups: Always test playbooks on a subset before rolling out to all hosts
4. Monitor Resources: Continuously check CPU, memory, and disk usage
5. Version Control: Keep all playbooks and scripts in Git
6. Documentation: Document playbook purposes and dependencies

---

🔗 Resources

• Ansible Documentation
• Ansible Community Guides
• Docker Documentation
• GCP VM Documentation

---

📝 License

This project is licensed under the MIT License - see LICENSE file for details.

---

🤝 Contributing

Contributions are welcome! Please submit pull requests or open issues for bugs and feature requests.

---

✍️ Author

Created as a comprehensive Ansible practice lab for DevOps engineers and infrastructure automation practitioners.

---

Last Updated: December 2025

Status: Production-Ready ✓