Market Data API

A production-ready microservice that fetches market data, processes it through a streaming pipeline, and serves it via REST APIs. Built with FastAPI, PostgreSQL, Kafka, and Redis with enterprise-grade monitoring, security, and operational capabilities.

🚀 Features

Core Functionality

• Real-time Market Data Processing: Kafka event streaming pipeline with producer/consumer architecture
• High-performance Caching: Redis for fast data access with intelligent cache invalidation
• Database Management: PostgreSQL with SQLAlchemy ORM and Alembic migrations
• API Gateway: FastAPI with automatic OpenAPI documentation

Advanced Monitoring & Observability

• Prometheus Metrics: Custom business metrics, HTTP metrics, database connection pools
• Grafana Dashboards: Pre-configured dashboards for API performance, database health, and system metrics
• Health Checks: Comprehensive health and readiness endpoints
• Audit Logging: Complete audit trail for security and compliance

Security & Compliance

• Rate Limiting: Distributed rate limiting with Redis backend
• Authentication: API key-based authentication system
• Input Validation: Pydantic schema validation with custom validators
• Audit Trail: Comprehensive logging of all API access and data operations
• CORS Support: Configurable cross-origin resource sharing

Operational Excellence

• Docker Containerization: Complete containerized deployment
• CI/CD Pipeline: GitHub Actions with automated testing and deployment
• Database Migrations: Alembic for schema versioning
• Error Handling: Graceful error handling with detailed logging
• Graceful Shutdown: Proper cleanup of connections and resources

Testing & Quality

• 95%+ Test Coverage: Unit, integration, API, and performance tests
• Code Quality: Linting with flake8, formatting with black, type checking with mypy
• Performance Testing: Load testing capabilities
• Postman Collection: Complete API testing suite with automated test scripts

🏗️ Architecture

graph TB
    A[FastAPI App] --> B[PostgreSQL]
    A --> C[Redis Cache]
    A --> D[Kafka Producer]
    D --> E[Kafka Topic: price-events]
    E --> F[Kafka Consumer]
    F --> G[Moving Average Calculator]
    G --> B
    A --> H[Prometheus]
    H --> I[Grafana]
    A --> J[Audit Logger]
    A --> K[Rate Limiter]
    C --> L[Cache Manager]
    M[Health Checker] --> A
    N[Error Handler] --> A

Loading

📊 Data Flow

sequenceDiagram
    participant Client
    participant API
    participant Cache
    participant DB
    participant Kafka
    participant Consumer
    participant Monitor

    Client->>API: GET /prices/latest?symbol=AAPL
    API->>Cache: Check cache
    alt Cache Hit
        Cache-->>API: Return cached price
    else Cache Miss
        API->>DB: Query latest price
        DB-->>API: Return price data
        API->>Cache: Store in cache
    end
    API->>Monitor: Log metrics
    API-->>Client: JSON response

    Client->>API: POST /prices/poll
    API->>Kafka: Publish price event
    Kafka->>Consumer: Consume event
    Consumer->>DB: Calculate & store moving average
    Consumer->>Monitor: Update metrics

Loading

🛠️ Tech Stack

Backend Framework

• FastAPI: Modern, fast web framework with automatic API documentation
• SQLAlchemy: Powerful ORM with connection pooling
• Alembic: Database migration management
• Pydantic: Data validation and serialization

Data Storage & Caching

• PostgreSQL: Primary database with advanced indexing
• Redis: High-performance caching and rate limiting
• Apache Kafka: Event streaming platform

Monitoring & Observability

• Prometheus: Metrics collection and storage
• Grafana: Visualization and alerting
• Custom Metrics: Business-specific monitoring

DevOps & Deployment

• Docker: Containerization
• Docker Compose: Multi-service orchestration
• GitHub Actions: CI/CD pipeline
• Health Checks: Service monitoring

Testing & Quality

• pytest: Testing framework
• coverage: Code coverage analysis
• Postman: API testing and documentation

📋 API Endpoints

Market Data Endpoints

GET /prices/latest?symbol={symbol}&provider={provider?}

Response:

{
  "symbol": "AAPL",
  "price": 150.25,
  "timestamp": "2024-03-20T10:30:00Z",
  "provider": "alpha_vantage"
}

Polling Job Management

POST /prices/poll
Content-Type: application/json

{
  "symbols": ["AAPL", "MSFT"],
  "interval": 60,
  "provider": "alpha_vantage"
}

Response (202 Accepted):

{
  "job_id": "poll_123",
  "status": "accepted",
  "config": {
    "symbols": ["AAPL", "MSFT"],
    "interval": 60
  }
}

Data Management

• GET /prices/ - List market data with pagination and filtering
• GET /prices/{symbol}/moving-average - Get moving average calculations
• GET /prices/symbols - List all tracked symbols
• POST /prices/ - Create new market data
• PUT /prices/{id} - Update market data
• DELETE /prices/{id} - Delete market data

System & Monitoring

• GET /health - Health check endpoint
• GET /ready - Readiness probe
• GET /metrics - Prometheus metrics
• GET /docs - Interactive API documentation

Job Management

• GET /prices/poll - List all polling jobs
• GET /prices/poll/{job_id} - Get job status
• DELETE /prices/poll/{job_id} - Delete polling job

🚀 Quick Start

Prerequisites

• Docker and Docker Compose
• Python 3.9+
• Git

1. Clone and Setup

git clone <your-repo-url>
cd market-data-api

2. Start All Services

# Start the complete stack
docker-compose up -d

# Or use the convenience script
./scripts/start_for_postman.sh

3. Install Dependencies

pip install -r requirements.txt

4. Run Database Migrations

alembic upgrade head

5. Start the Application

python -m app.main

6. Verify Setup

# Health check
curl http://localhost:8000/health

# API documentation
open http://localhost:8000/docs

# Get latest price
curl "http://localhost:8000/prices/latest?symbol=AAPL"

🧪 Testing

Run All Tests

pytest tests/ --cov=app --cov-report=term-missing

Test Categories

• Unit Tests: Core business logic and service functions
• Integration Tests: Database and external service interactions
• API Tests: Endpoint functionality and response validation
• Performance Tests: Load testing and performance benchmarks

Test Coverage

• 95%+ Coverage: Comprehensive test coverage across all modules
• Automated Testing: CI/CD pipeline with automated test execution
• Postman Collection: Complete API testing suite with test scripts

📊 Monitoring & Observability

Prometheus Metrics

• HTTP Metrics: Request rates, latencies, status codes
• Database Metrics: Connection pool status, query performance
• Redis Metrics: Cache hit/miss ratios, memory usage
• Kafka Metrics: Producer/consumer lag, message rates
• Business Metrics: Market data points processed, symbols tracked

Grafana Dashboards

• API Performance Dashboard: Request rates, response times, error rates
• Database Performance Dashboard: Query performance, connection pools
• System Resources Dashboard: CPU, memory, disk usage
• Business Metrics Dashboard: Data processing rates, cache performance

Health Monitoring

• Health Checks: Service health status
• Readiness Probes: Service readiness for traffic
• Liveness Probes: Service liveness detection

🔒 Security Features

Rate Limiting

• Distributed Rate Limiting: Redis-backed rate limiting
• Per-Endpoint Limits: Configurable limits per API endpoint
• IP-Based Limiting: Client IP-based rate limiting
• Graceful Degradation: Fail-open on Redis errors

Authentication & Authorization

• API Key Authentication: Secure API key-based authentication
• Permission Levels: Read, write, and admin permissions
• Audit Logging: Complete authentication event logging

Input Validation

• Pydantic Schemas: Strong type validation
• Custom Validators: Business-specific validation rules
• SQL Injection Protection: Parameterized queries

Audit & Compliance

• Comprehensive Logging: All API access logged
• Security Events: Security violation tracking
• Data Access Logging: All data operations logged
• Compliance Ready: Audit trail for regulatory compliance

🐳 Docker Deployment

Single Container

docker build -t market-data-api .
docker run -p 8000:8000 market-data-api

Multi-Service Stack

docker-compose up -d

Services Included

• API Service: FastAPI application
• PostgreSQL: Primary database
• Redis: Caching and rate limiting
• Prometheus: Metrics collection
• Grafana: Monitoring dashboards

📁 Project Structure

market-data-api/
├── app/
│   ├── api/              # API endpoints and routing
│   ├── core/             # Core functionality (auth, config, rate limiting, audit)
│   ├── db/               # Database configuration and session management
│   ├── models/           # SQLAlchemy ORM models
│   ├── schemas/          # Pydantic data schemas
│   └── services/         # Business logic services (market data, Kafka, Redis)
├── tests/                # Comprehensive test suite
├── scripts/              # Operational scripts
├── alembic/              # Database migrations
├── docker-compose.yml    # Multi-service orchestration
├── Dockerfile           # Container definition
├── prometheus.yml       # Prometheus configuration
├── grafana_dashboard.json # Grafana dashboard definitions
├── postman_collection.json # API testing collection
└── requirements.txt     # Python dependencies

🔧 Configuration

Environment Variables

All configuration is handled through environment variables with sensible defaults. Copy .env.example to .env and customize as needed:

# Copy the example configuration
cp .env.example .env

# Edit the configuration
nano .env

Key Configuration Categories

API Configuration

PROJECT_NAME=Market Data Service
DEBUG=false
HOST=0.0.0.0
PORT=8000

Database Configuration

DATABASE_URL=postgresql://user:pass@localhost/marketdata
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=market_data

Redis Configuration

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
CACHE_TTL=300
CACHE_ENABLED=true

Kafka Configuration

KAFKA_BOOTSTRAP_SERVERS=localhost:9092
KAFKA_CONSUMER_GROUP=market_data_group
KAFKA_TOPIC=price-events

Security Configuration

API_KEY=your-api-key-here
SECRET_KEY=your-secret-key-here
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=60

Monitoring Configuration

PROMETHEUS_ENABLED=true
GRAFANA_ENABLED=true
LOG_LEVEL=INFO

Dynamic Configuration Features

• No Hardcoded Values: All configuration is externalized
• Environment-Specific: Different configs for dev/staging/prod
• Validation: Pydantic settings validation with type checking
• Defaults: Sensible defaults for all settings
• Hot Reload: Configuration changes without restart (where applicable)

Configuration Management

• Environment-based: Different configs for dev/staging/prod
• Validation: Pydantic settings validation with comprehensive error messages
• Defaults: Sensible defaults for all settings
• Documentation: All settings documented with descriptions
• Type Safety: Strong typing for all configuration values

📈 Performance Characteristics

Response Times

• Cached Data: < 50ms response time
• Database Queries: < 100ms for indexed queries
• Kafka Operations: < 10ms for message publishing

Throughput

• API Requests: 1000+ requests/second
• Cache Hit Rate: > 90% for frequently accessed data
• Database: Optimized queries with proper indexing

Scalability

• Horizontal Scaling: Stateless API design
• Database Scaling: Connection pooling and query optimization
• Cache Scaling: Redis cluster support

🚨 Troubleshooting

Common Issues

1. Database Connection Failed

   # Check PostgreSQL status
   docker-compose ps db
   
   # Check logs
   docker-compose logs db
   
   # Verify connection string
   echo $DATABASE_URL

2. Redis Connection Issues

   # Check Redis status
   docker-compose ps redis
   
   # Test Redis connection
   docker-compose exec redis redis-cli ping

3. Kafka Connection Issues

   # Check Kafka status
   docker-compose ps kafka
   
   # Check Kafka logs
   docker-compose logs kafka

Monitoring & Debugging

# Application logs
docker-compose logs -f api

# Database logs
docker-compose logs -f db

# Redis logs
docker-compose logs -f redis

# Kafka logs
docker-compose logs -f kafka

# Prometheus metrics
curl http://localhost:9090/api/v1/targets

# Grafana dashboard
open http://localhost:3000

Performance Tuning

• Database Indexing: Ensure proper indexes on frequently queried columns
• Cache Strategy: Optimize cache TTL and invalidation
• Connection Pooling: Tune database connection pool settings
• Rate Limiting: Adjust rate limits based on usage patterns

📝 Development

Code Quality Standards

• Linting: flake8 for code style enforcement
• Formatting: black for consistent code formatting
• Type Checking: mypy for static type analysis
• Pre-commit Hooks: Automated code quality checks

Development Workflow

1. Create feature branch from main
2. Implement functionality with tests
3. Run full test suite
4. Update documentation
5. Submit pull request with comprehensive description

Adding New Features

• API Endpoints: Add to appropriate router in app/api/
• Database Models: Create models in app/models/
• Business Logic: Implement in app/services/
• Tests: Add corresponding tests in tests/

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes with tests
4. Ensure all tests pass
5. Update documentation
6. Submit a pull request

---

Enterprise-Grade Market Data Microservice with Full Observability, Security, and Operational Excellence