Unified API gateway with authentication, rate limiting, and observability
Unified API Gateway for North Shore AI services, providing authentication, rate limiting, request routing, and telemetry.
NSAI Gateway is a high-performance API gateway built with Elixir that sits between clients and backend NSAI services (Work, Forge, Anvil, Crucible). It provides:
- Authentication: API Key and JWT token support
- Authorization: Tenant isolation and access control
- Rate Limiting: Per-tenant and per-endpoint limits with sliding window algorithm
- Request Routing: Intelligent routing to backend services
- Service Discovery: Health checking and failover
- Telemetry: Comprehensive metrics and monitoring
- Load Balancing: Request distribution with retry logic
┌─────────┐
│ Clients │
└────┬────┘
│
▼
┌─────────────────┐
│ NSAI Gateway │
│ │
│ ┌───────────┐ │
│ │ Auth │ │
│ ├───────────┤ │
│ │ Rate Limit│ │
│ ├───────────┤ │
│ │ Router │ │
│ ├───────────┤ │
│ │ Proxy │ │
│ └───────────┘ │
└────┬─┬─┬─┬──────┘
│ │ │ │
▼ ▼ ▼ ▼
┌────┐┌─────┐┌───────┐┌──────────┐
│Work││Forge││Anvil ││Crucible │
└────┘└─────┘└───────┘└──────────┘
Add nsai_gateway to your list of dependencies in mix.exs:
def deps do
[
{:nsai_gateway, "~> 0.1.0"}
]
endConfigure the gateway in config/config.exs:
config :nsai_gateway,
port: 4000,
jwt_secret: System.get_env("JWT_SECRET"),
tenant_rate_limit: 1000, # requests per minute
endpoint_rate_limits: %{
"jobs" => 100,
"samples" => 200,
"labels" => 150,
"experiments" => 50
}
# Backend service URLs
config :nsai_gateway, :services, %{
"work" => "http://localhost:4001",
"forge" => "http://localhost:4002",
"anvil" => "http://localhost:4003",
"crucible" => "http://localhost:4004"
}
# API Keys (use database in production)
config :nsai_gateway, :api_keys, %{
"your-api-key-here" => "tenant-name"
}PORT: HTTP server port (default: 4000)JWT_SECRET: Secret key for JWT signingWORK_SERVICE_URL: Work service backend URLFORGE_SERVICE_URL: Forge service backend URLANVIL_SERVICE_URL: Anvil service backend URLCRUCIBLE_SERVICE_URL: Crucible service backend URL
# Fetch dependencies
mix deps.get
# Compile
mix compile
# Run in development
mix run --no-halt
# Or with iex
iex -S mixAll API routes are prefixed with /api/v1/{service}/:
/api/v1/jobs/* -> Work service
/api/v1/samples/* -> Forge service
/api/v1/labels/* -> Anvil service
/api/v1/experiments/* -> Crucible service
Include API key in the Authorization header:
curl -H "Authorization: ApiKey your-api-key-here" \
http://localhost:4000/api/v1/jobsOr as a query parameter:
curl http://localhost:4000/api/v1/jobs?api_key=your-api-key-hereGenerate a token:
{:ok, token, claims} = NsaiGateway.Auth.JWT.generate("tenant-name", "user-id", 3600)Use the token:
curl -H "Authorization: Bearer eyJhbGc..." \
http://localhost:4000/api/v1/jobscurl http://localhost:4000/health
# Response:
# {"status": "healthy", "service": "nsai_gateway"}The gateway implements a sliding window rate limiter with two levels:
- Tenant-level: Global limit per tenant (default: 1000 req/min)
- Endpoint-level: Per-service limits (configurable)
When rate limited, responses include:
HTTP/1.1 429 Too Many Requests
Retry-After: 60
{
"error": "Rate Limit Exceeded",
"message": "Too many requests. Please try again later."
}
The gateway emits telemetry events for monitoring:
[:nsai_gateway, :proxy, :success]- Successful proxy requests[:nsai_gateway, :proxy, :error]- Failed proxy requests[:nsai_gateway, :auth, :success]- Successful authentication[:nsai_gateway, :auth, :failure]- Failed authentication[:nsai_gateway, :rate_limit, :exceeded]- Rate limit violations
NsaiGateway.Telemetry.attach_handlers()# Run all tests
mix test
# Run with coverage
mix test --cover
# Run specific test file
mix test test/nsai_gateway/auth_test.exs# Format code
mix format
# Run Credo (if added)
mix credo --strict
# Run Dialyzer (if added)
mix dialyzer- HTTP Request arrives at the gateway
- Router matches the path to a backend service
- Authentication validates API key or JWT token
- Rate Limiter checks tenant and endpoint limits
- Proxy forwards request to backend service with retry logic
- Response is returned to the client
- Telemetry events are emitted for monitoring
- Plug-based HTTP router
- Route matching and dispatch
- Health check endpoint
- HTTP request forwarding
- Service discovery integration
- Retry with exponential backoff
- Response streaming
- API key validation
- JWT token verification
- Tenant assignment
- Sliding window algorithm (Hammer)
- Per-tenant limits
- Per-endpoint limits
- Service discovery
- Health checking
- Failover support
- Metrics collection
- Event emission
- Performance monitoring
- JWT Secret: Use strong, random secrets in production
- API Keys: Store in secure database, not config files
- HTTPS: Always use TLS in production
- Rate Limiting: Tune limits based on capacity
- Connection Pooling: Configure HTTP client pools
- Caching: Add caching layer for service discovery
- Load Balancing: Use multiple gateway instances
- Monitoring: Set up proper telemetry collection
- Horizontal Scaling: Run multiple gateway instances
- Service Discovery: Integrate with nsai_registry
- Circuit Breakers: Add circuit breaker pattern
- Async Processing: Use message queues for long operations
- Integration with nsai_registry for dynamic service discovery
- Circuit breaker pattern for backend failures
- Request/response transformation hooks
- GraphQL gateway support
- WebSocket proxying
- API versioning strategies
- Response caching
- Request validation schemas
See the main North-Shore-AI repository for contribution guidelines.
MIT License - see LICENSE file for details.
- nsai_registry - Service discovery and registration
- crucible_framework - ML experimentation framework
- cns - Critic-Network Synthesis
For issues and questions:
- GitHub Issues: https://github.com/North-Shore-AI/nsai_gateway/issues
- Documentation: https://hexdocs.pm/nsai_gateway