LocalStack Local Development

This guide explains how to set up and use LocalStack for local development and testing of Redshift Spectra.

Overview

LocalStack provides a fully functional local AWS cloud stack that allows you to develop and test cloud applications offline. This project uses Terragrunt to manage infrastructure in both LocalStack (local) and AWS (dev/prod) environments, ensuring consistency across all deployments.

!!! warning "LocalStack Pro Required for Full Functionality"

**Redshift Spectra requires LocalStack Pro** for complete local development experience.

The following critical features are **only available in LocalStack Pro**:

| Feature | Community | Pro | Impact on Redshift Spectra |
|---------|:---------:|:---:|----------------------------|
| **Redshift Data API** | ❌ | ✅ | Core query execution functionality |
| **Lambda Layers** | ❌ | ✅ | Shared dependency management |
| IAM Policy Enforcement | ❌ | ✅ | Security testing |
| CloudWatch Metrics | ❌ | ✅ | Monitoring and alerting |
| Persistence | ❌ | ✅ | Data retention across restarts |

**Without LocalStack Pro**, you can still:

- Deploy and test basic infrastructure (S3, DynamoDB, IAM roles)
- Run unit tests with mocked AWS services
- Validate Terraform/Terragrunt configurations

**With LocalStack Pro**, you get:

- Full end-to-end integration testing
- Redshift Data API simulation for query testing
- Lambda Layers for optimized function deployment
- Complete production-like environment

Get LocalStack Pro at: [https://localstack.cloud/pricing/](https://localstack.cloud/pricing/)

Prerequisites

Docker: LocalStack runs as a Docker container
Terraform: >= 1.11.0
Terragrunt: >= 0.99.0
AWS CLI (optional): For testing with awslocal or direct commands

Quick Start

1. Start LocalStack

# Using Task
task local:start

# Or using Docker Compose directly
docker compose up -d localstack

# Or using the helper script (Bash)
./scripts/localstack-deploy.sh start

2. Deploy Infrastructure

# Deploy all modules to LocalStack
task local:deploy

# Or step by step
task local:tg-init
task local:tg-plan
task local:tg-apply

3. Verify Deployment

# Check LocalStack status
task local:status

# List created resources
aws --endpoint-url=http://localhost:4566 s3 ls
aws --endpoint-url=http://localhost:4566 dynamodb list-tables
aws --endpoint-url=http://localhost:4566 secretsmanager list-secrets

Architecture

The following diagram illustrates how LocalStack integrates with the development workflow, using the same Terraform modules across all environments:

flowchart TB
    subgraph Environments["🌍 Deployment Environments"]
        direction LR
        LOCAL["🏠 LocalStack<br/><code>localhost:4566</code><br/><i>Account: 000000000000</i>"]
        DEV["🔧 AWS Dev<br/><code>dev.aws.amazon.com</code><br/><i>Account: 123456789012</i>"]
        PROD["🚀 AWS Prod<br/><code>prod.aws.amazon.com</code><br/><i>Account: 987654321098</i>"]
    end

    subgraph TerragruntConfig["📁 Terragrunt Configuration"]
        direction TB
        ROOT_AWS["terragrunt.hcl<br/><i>AWS Root Config</i>"]
        ROOT_LOCAL["terragrunt-local.hcl<br/><i>LocalStack Root Config</i>"]
        COMMON["common.hcl<br/><i>Shared Variables</i>"]

        subgraph EnvConfigs["environments/"]
            ENV_LOCAL["local/<br/>• account.hcl<br/>• us-east-1/env.hcl"]
            ENV_DEV["dev/<br/>• account.hcl<br/>• us-east-1/env.hcl"]
            ENV_PROD["prod/<br/>• account.hcl<br/>• us-east-1/env.hcl"]
        end
    end

    subgraph TerraformModules["🧱 Terraform Modules<br/><i>Shared across all environments</i>"]
        direction LR
        MOD_DDB[("💾 dynamodb")]
        MOD_S3[("📦 s3")]
        MOD_IAM[("🔐 iam")]
        MOD_LAMBDA[("⚡ lambda")]
        MOD_APIGW[("🌐 api-gateway")]
        MOD_MON[("📊 monitoring")]
    end

    subgraph LocalStackServices["🐳 LocalStack Container"]
        direction LR
        LS_S3["S3"]
        LS_DDB["DynamoDB"]
        LS_IAM["IAM"]
        LS_SM["Secrets<br/>Manager"]
        LS_LAMBDA["Lambda"]
        LS_APIGW["API<br/>Gateway"]
        LS_CW["CloudWatch"]
    end

    %% Connections
    ROOT_LOCAL --> ENV_LOCAL
    ROOT_AWS --> ENV_DEV
    ROOT_AWS --> ENV_PROD
    COMMON --> ROOT_LOCAL
    COMMON --> ROOT_AWS

    ENV_LOCAL --> TerraformModules
    ENV_DEV --> TerraformModules
    ENV_PROD --> TerraformModules

    TerraformModules --> LOCAL
    TerraformModules --> DEV
    TerraformModules --> PROD

    LOCAL --> LocalStackServices

    %% Styling
    classDef localEnv fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    classDef devEnv fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    classDef prodEnv fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    classDef config fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    classDef modules fill:#fce4ec,stroke:#c2185b,stroke-width:2px
    classDef services fill:#e0f2f1,stroke:#00796b,stroke-width:2px

    class LOCAL localEnv
    class DEV devEnv
    class PROD prodEnv
    class ROOT_AWS,ROOT_LOCAL,COMMON,ENV_LOCAL,ENV_DEV,ENV_PROD config
    class MOD_DDB,MOD_S3,MOD_IAM,MOD_LAMBDA,MOD_APIGW,MOD_MON modules
    class LS_S3,LS_DDB,LS_IAM,LS_SM,LS_LAMBDA,LS_APIGW,LS_CW services

Development Workflow

sequenceDiagram
    autonumber
    participant Dev as 👨‍💻 Developer
    participant Docker as 🐳 Docker
    participant LS as 📦 LocalStack
    participant TG as 🔧 Terragrunt
    participant TF as 📐 Terraform
    participant AWS as ☁️ AWS Services

    Note over Dev,AWS: Local Development Phase

    Dev->>Docker: task local:start
    Docker->>LS: Start container (port 4566)
    LS-->>Docker: Ready ✓
    Docker-->>Dev: LocalStack running

    Dev->>TG: task local:tg-apply
    TG->>TG: Load terragrunt-local.hcl
    TG->>TF: Generate provider.tf<br/>(LocalStack endpoints)
    TF->>LS: Create resources<br/>(S3, DynamoDB, IAM...)
    LS-->>TF: Resources created ✓
    TF-->>TG: Apply complete
    TG-->>Dev: Infrastructure deployed locally

    Note over Dev,AWS: Testing & Iteration

    Dev->>LS: Run integration tests
    LS-->>Dev: Test results
    Dev->>Dev: Fix issues, iterate

    Note over Dev,AWS: Cloud Deployment Phase

    Dev->>TG: task infra:apply-dev
    TG->>TG: Load terragrunt.hcl
    TG->>TF: Generate provider.tf<br/>(AWS endpoints)
    TF->>AWS: Create resources
    AWS-->>TF: Resources created ✓
    TF-->>TG: Apply complete
    TG-->>Dev: Deployed to AWS Dev ✓

Environment Comparison

graph LR
    subgraph Local["🏠 LocalStack Environment"]
        L_CMD["task local:deploy"]
        L_CONFIG["terragrunt-local.hcl"]
        L_BACKEND["Local State<br/><code>terraform.tfstate</code>"]
        L_ENDPOINT["http://localhost:4566"]
        L_CREDS["AWS_ACCESS_KEY_ID=test<br/>AWS_SECRET_ACCESS_KEY=test"]
    end

    subgraph Cloud["☁️ AWS Environment"]
        C_CMD["task deploy:dev"]
        C_CONFIG["terragrunt.hcl"]
        C_BACKEND["S3 Remote State<br/><code>s3://...-terraform-state</code>"]
        C_ENDPOINT["AWS Regional Endpoints"]
        C_CREDS["IAM Credentials<br/><code>aws configure</code>"]
    end

    L_CMD --> L_CONFIG --> L_BACKEND
    L_CONFIG --> L_ENDPOINT
    L_CONFIG --> L_CREDS

    C_CMD --> C_CONFIG --> C_BACKEND
    C_CONFIG --> C_ENDPOINT
    C_CONFIG --> C_CREDS

    style Local fill:#e3f2fd,stroke:#1976d2
    style Cloud fill:#fff8e1,stroke:#ffa000

Available Commands

Taskfile Targets

Command	Description
`task local:start`	Start LocalStack container
`task local:stop`	Stop LocalStack container
`task local:status`	Check LocalStack health status
`task local:logs`	Stream LocalStack container logs
`task local:reset`	Reset LocalStack (destroy volumes and restart)
`task local:deploy`	Full local deployment (start + apply)
`task local:tg-init`	Initialize Terragrunt for LocalStack
`task local:tg-plan`	Plan all changes for LocalStack
`task local:tg-apply`	Apply all changes to LocalStack
`task local:tg-destroy`	Destroy all LocalStack resources
`task local:tg-output`	Show Terragrunt outputs
`task local:tg-graph`	Show dependency graph for LocalStack

Module-Specific Commands

Command	Description
`task local:plan-dynamodb`	Plan DynamoDB changes
`task local:apply-dynamodb`	Apply DynamoDB changes
`task local:plan-s3`	Plan S3 changes
`task local:apply-s3`	Apply S3 changes
`task local:plan-iam`	Plan IAM changes
`task local:apply-iam`	Apply IAM changes

Formatting Commands

Command	Description
`task infra:tf-fmt`	Format all Terraform files
`task infra:tf-fmt-check`	Check Terraform formatting (no changes)
`task infra:tg-fmt`	Format all Terragrunt HCL files
`task infra:tg-fmt-check`	Check Terragrunt formatting (no changes)
`task infra:iac-fmt`	Format all Terraform and Terragrunt files
`task infra:iac-fmt-check`	Check all IaC formatting

Environment Switching

The project supports seamless switching between local and AWS environments:

Local Development (LocalStack)

# Uses terragrunt-local.hcl configuration
cd terragrunt/environments/local/us-east-1

# Set LocalStack credentials
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Run with LocalStack config
terragrunt run-all apply --terragrunt-config ../../terragrunt-local.hcl

AWS Development

# Uses standard terragrunt.hcl configuration
cd terragrunt/environments/dev/us-east-1

# Use your AWS credentials
aws configure

# Run with AWS config (default)
terragrunt run-all apply

AWS Production

cd terragrunt/environments/prod/us-east-1
terragrunt run-all apply

LocalStack Services

This project uses the following AWS services emulated by LocalStack:

Service	Community	Pro	Notes
S3	✅	✅	Object storage for query results
DynamoDB	✅	✅	Job state and session management
IAM	✅	✅	Roles and policies (permissive mode in Community)
Secrets Manager	✅	✅	JWT secrets storage
Lambda	✅	✅	Function execution (requires Docker)
Lambda Layers	❌	✅	Shared dependencies - Pro only
API Gateway	✅	✅	REST API endpoints
CloudWatch Logs	✅	✅	Logging
STS	✅	✅	Identity
Redshift Data API	❌	✅	Query execution - Pro only

!!! tip "Running without Pro"

If you don't have LocalStack Pro, you can still develop locally by:

1. Using mocked services in unit tests (already configured)
2. Deploying Lambda functions as "fat" packages (dependencies bundled):
   ```bash
   # Build fat packages for LocalStack Community
   task build:lambda-fat
   ```
   See [Fat Lambda Packages](../deployment/lambda-layer.md#fat-lambda-packages-localstack-community) for details.
3. Testing against a real AWS dev environment for Redshift queries

Testing with LocalStack

Using AWS CLI with LocalStack

# Configure alias for awslocal
alias awslocal='aws --endpoint-url=http://localhost:4566'

# Or use environment variable
export AWS_ENDPOINT_URL=http://localhost:4566

# Examples
awslocal s3 ls
awslocal dynamodb list-tables
awslocal secretsmanager list-secrets

Using Python SDK (boto3)

import boto3

# Create client pointing to LocalStack
dynamodb = boto3.client(
    'dynamodb',
    endpoint_url='http://localhost:4566',
    region_name='us-east-1',
    aws_access_key_id='test',
    aws_secret_access_key='test'
)

# Use normally
response = dynamodb.list_tables()
print(response['TableNames'])

Running Integration Tests

# Start LocalStack first
task local:start

# Deploy infrastructure
task local:deploy

# Run integration tests against LocalStack
LOCALSTACK_ENDPOINT=http://localhost:4566 pytest tests/integration/

Configuration

Environment Variables

The project uses a unified .env.example file that works for both LocalStack and AWS environments:

# For LocalStack (defaults work out of the box)
cp .env.example .env

# For AWS (update with your actual values)
cp .env.example .env
# Edit .env and update:
# - ENVIRONMENT=dev
# - SPECTRA_REDSHIFT_CLUSTER_ID=your-cluster
# - SPECTRA_REDSHIFT_SECRET_ARN=arn:aws:...
# - etc.

Key environment variables for LocalStack:

Variable	Default	Description
`ENVIRONMENT`	`local`	Environment name (local/dev/prod)
`LOCALSTACK_ENDPOINT`	`http://localhost:4566`	LocalStack endpoint URL
`AWS_ACCESS_KEY_ID`	`test`	AWS credentials (any value works for LocalStack)
`AWS_SECRET_ACCESS_KEY`	`test`	AWS credentials (any value works for LocalStack)
`LAMBDA_EXECUTOR`	`docker`	Lambda execution mode
`LOCALSTACK_PERSISTENCE`	`0`	Persist data across restarts

Docker Compose Configuration

The docker-compose.yml file configures LocalStack:

services:
  localstack:
    image: localstack/localstack:latest
    ports:
      - "127.0.0.1:4566:4566"
    environment:
      - SERVICES=s3,dynamodb,lambda,iam,apigateway,secretsmanager,cloudwatch,logs,sts
      - LAMBDA_EXECUTOR=docker
      - LAMBDA_DOCKER_NETWORK=redshift-spectra-network
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - localstack-data:/var/lib/localstack

Terragrunt Configuration

LocalStack environment configuration is located at:

terragrunt/environments/local/account.hcl - Account settings
terragrunt/environments/local/us-east-1/env.hcl - Environment settings
terragrunt/environments/local/us-east-1/region.hcl - Region and endpoint settings

Persistence

By default, LocalStack data is persisted in a Docker volume (localstack-data). To reset:

task local:reset
# or
docker compose down -v && docker compose up -d localstack

Troubleshooting

LocalStack Not Starting

# Check Docker is running
docker info

# Check port availability
netstat -an | grep 4566

# View container logs
docker compose logs localstack

Terraform State Issues

The local environment uses local state files (not S3) to avoid chicken-and-egg problems:

# State files are stored at:
# terragrunt/environments/local/us-east-1/<module>/terraform.tfstate

# To reset state, remove these files:
find terragrunt/environments/local -name "terraform.tfstate*" -delete

Lambda Execution Issues

LocalStack Lambda requires Docker access:

# Ensure Docker socket is mounted
docker inspect redshift-spectra-localstack | grep -A5 "Binds"

# Check Lambda executor setting
docker exec redshift-spectra-localstack env | grep LAMBDA

Best Practices

Always start with local: Develop and test locally before deploying to AWS
Use consistent configuration: Same Terraform modules across all environments
Reset regularly: Clear LocalStack data when switching feature branches
Test infrastructure changes locally: Run task local:tg-plan before task infra:plan-dev
Use LocalStack for integration tests: Faster and cheaper than real AWS

Navigation Menu

LocalStack Local Development

Overview

Prerequisites

Quick Start

1. Start LocalStack

2. Deploy Infrastructure

3. Verify Deployment

Architecture

Development Workflow

Environment Comparison

Available Commands

Taskfile Targets

Module-Specific Commands

Formatting Commands

Environment Switching

Local Development (LocalStack)

AWS Development

AWS Production

LocalStack Services

Testing with LocalStack

Using AWS CLI with LocalStack

Using Python SDK (boto3)

Running Integration Tests

Configuration

Environment Variables

Docker Compose Configuration

Terragrunt Configuration

Persistence

Troubleshooting

LocalStack Not Starting

Terraform State Issues

Lambda Execution Issues

Best Practices

Related Documentation

FilesExpand file tree

localstack.md

Latest commit

History

localstack.md

File metadata and controls

LocalStack Local Development

Overview

Prerequisites

Quick Start

1. Start LocalStack

2. Deploy Infrastructure

3. Verify Deployment

Architecture

Development Workflow

Environment Comparison

Available Commands

Taskfile Targets

Module-Specific Commands

Formatting Commands

Environment Switching

Local Development (LocalStack)

AWS Development

AWS Production

LocalStack Services

Testing with LocalStack

Using AWS CLI with LocalStack

Using Python SDK (boto3)

Running Integration Tests

Configuration

Environment Variables

Docker Compose Configuration

Terragrunt Configuration

Persistence

Troubleshooting

LocalStack Not Starting

Terraform State Issues

Lambda Execution Issues

Best Practices

Related Documentation