Enterprise CI/CD Platform

A production-grade, end-to-end CI/CD platform demonstrating how enterprise engineering teams build, test, secure, and deploy Java microservices to Kubernetes. Every tool in the stack has a deliberate purpose, and this README explains not just what each tool does but why it belongs in a mature pipeline.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────────┐
│                      ENTERPRISE CI/CD PLATFORM ARCHITECTURE                     │
│                                                                                 │
│  DEVELOPER                SOURCE CONTROL          CI/CD ENGINE                  │
│  ┌─────────┐              ┌──────────────┐        ┌──────────────────────────┐ │
│  │  Local  │  git push    │   GitHub     │ webhook │       Jenkins            │ │
│  │  Dev    │ ──────────►  │  Repository  │ ──────► │  Declarative Pipeline   │ │
│  │  IDE    │  Pull Request│  - Branches  │        │  K8s Pod Agents          │ │
│  └─────────┘              │  - Webhooks  │        │  21-stage pipeline       │ │
│                           │  - PRs       │        └────────────┬─────────────┘ │
│                           └──────────────┘                     │               │
│                                                                 │               │
│  ─────────────────────────── CI STAGES ──────────────────────────────────────  │
│                                                                 │               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐      │               │
│  │  Maven   │  │SonarQube │  │  OWASP   │  │  Maven   │      │               │
│  │  Build   │  │  SAST +  │  │  Dep     │  │  Deploy  │      │               │
│  │  + Test  │  │  Quality │  │  Check   │  │  → Nexus │      │               │
│  │  JaCoCo  │  │  Gate    │  │  CVE     │  │  Artifact│      │               │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘      │               │
│                                                                 │               │
│  ─────────────────────────── CD STAGES ──────────────────────────────────────  │
│                                                                 │               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐      │               │
│  │  Docker  │  │  Trivy   │  │  Push    │  │  Ansible │      │               │
│  │  Multi-  │  │Container │  │  Image   │  │  Pre-    │      │               │
│  │  Stage   │  │  CVE     │  │  to      │  │  Deploy  │      │               │
│  │  Build   │  │  Scan    │  │  Nexus   │  │  Config  │      │               │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘      │               │
│                                                                 │               │
│  ─────────────────────────── DEPLOY STAGES ─────────────────────────────────  │
│                                                                 │               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐      │               │
│  │  Helm    │  │  Smoke   │  │  Manual  │  │  Helm    │      │               │
│  │  Deploy  │  │  Tests   │  │  Approval│  │  Deploy  │      │               │
│  │  Dev     │  │  Dev     │  │  Gate    │  │  Staging │      │               │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘      │               │
│                                                                 │               │
│  ┌──────────────────────────────────────────────────────┐      │               │
│  │               KUBERNETES (EKS) CLUSTER               │      │               │
│  │                                                      │      │               │
│  │  ┌─────────────────────────────────────────────┐   │      │               │
│  │  │  Terraform — Provisions the Cluster         │   │      │               │
│  │  │  VPC + Subnets + EKS + ECR + IAM + KMS      │   │      │               │
│  │  └─────────────────────────────────────────────┘   │      │               │
│  │                                                      │      │               │
│  │  Dev Namespace    Staging Namespace   Prod Namespace │      │               │
│  │  1 replica        2 replicas          3+ replicas   │      │               │
│  │  (no HPA)         HPA 2-10            HPA 3-20       │      │               │
│  │                                                      │      │               │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │      │               │
│  │  │  Pods    │  │  Helm    │  │  Ansible         │  │      │               │
│  │  │  Health  │  │  Chart   │  │  ConfigMaps      │  │      │               │
│  │  │  Probes  │  │  Values  │  │  ServiceMonitors │  │      │               │
│  │  └──────────┘  └──────────┘  └──────────────────┘  │      │               │
│  └──────────────────────────────────────────────────────┘      │               │
│                                                                 │               │
│  ─────────────────────────── OBSERVABILITY ─────────────────────────────────  │
│                                                                 │               │
│  ┌───────────────────────────────────┐  ┌───────────────────┐ │               │
│  │           Prometheus              │  │      Grafana       │ │               │
│  │  - Spring Boot /actuator/metrics  │  │  - Dashboard       │ │               │
│  │  - K8s node/pod metrics           │  │  - Alerts          │ │               │
│  │  - Jenkins metrics                │  │  - Deploy          │ │               │
│  │  - SonarQube metrics              │  │    Annotations     │ │               │
│  │  - Nexus metrics                  │  │  - Slack alerts    │ │               │
│  └───────────────────────────────────┘  └───────────────────┘ │               │
└─────────────────────────────────────────────────────────────────────────────────┘

Why This Stack

Every tool was chosen deliberately. Here is the rationale:

GitHub is the industry-standard source control platform. Webhook integration with Jenkins means every push triggers the pipeline within seconds — no manual build triggers.

Jenkins with Kubernetes pod agents scales dynamically — a new pod spins up for each build, uses its compute, then disappears. No idle agents burning money. The declarative pipeline syntax is version-controlled alongside the application code, so pipeline changes go through PR review.

Maven provides reproducible Java builds. The pom.xml pins every dependency version. The distributionManagement section routes artifacts to Nexus, meaning every JAR the pipeline produces is stored, versioned, and retrievable.

SonarQube catches bugs, security vulnerabilities, and code smells before code reaches production. The Quality Gate is the key feature — the pipeline cannot advance if the gate fails. This is how you enforce quality standards at scale without relying on human reviewers to catch everything.

Nexus is the artifact repository. Dependencies are proxied through Nexus (so mvn install never calls Maven Central directly), and built JARs are published to Nexus after a successful build. This means you can reproduce any release years later by pointing at the Nexus artifact, regardless of what upstream has changed.

Docker packages the application as an immutable, portable container. The multi-stage build keeps build tools out of the runtime image. Distroless base images eliminate an entire category of CVEs by removing the shell and package manager.

Trivy scans the built container image for CVEs before it is pushed to the registry. A CRITICAL vulnerability blocks the pipeline — you cannot push a vulnerable image to production.

Kubernetes provides self-healing, horizontal scaling, and zero-downtime rolling deployments. The HPA automatically scales replicas based on CPU and memory utilization. PodDisruptionBudgets ensure availability during node drains.

Terraform provisions the infrastructure. The EKS cluster, VPC, IAM roles, ECR registry, and KMS keys are all defined as code. Infrastructure changes go through the same Git PR process as application changes.

Ansible handles configuration management — things that happen between "infrastructure exists" and "application is deployed." It applies Kubernetes ConfigMaps, creates ServiceMonitors, verifies cluster health, and configures environment-specific settings. Ansible bridges the gap between Terraform (infrastructure) and Helm (application deployment).

Prometheus collects metrics from every component: the application (via Spring Boot Actuator /actuator/prometheus), Kubernetes nodes and pods, Jenkins build metrics, SonarQube quality metrics, and Nexus repository metrics. All metrics flow into a single Prometheus instance.

Grafana visualizes those metrics. The Jenkins pipeline creates a Grafana annotation on every production deployment, so you can instantly correlate "when did error rate spike?" with "what was deployed at that time?"

Repository Structure

enterprise-cicd-platform/
│
├── README.md                              # This file
├── docker/
│   ├── Dockerfile                         # Multi-stage, distroless, non-root
│   └── docker-compose.yml                 # Local tool stack (Jenkins, Sonar, Nexus, etc.)
│
├── app/                                   # Spring Boot application
│   ├── pom.xml                            # Maven: JaCoCo, OWASP, Nexus distribution
│   └── src/
│       ├── main/java/com/enterprise/app/
│       │   ├── Application.java
│       │   ├── controller/ProductController.java
│       │   ├── service/ProductService.java    # Prometheus metrics (Micrometer)
│       │   ├── model/Product.java             # Jakarta Bean Validation
│       │   ├── health/AppHealthIndicator.java
│       │   └── config/SecurityConfig.java     # Spring Security + security headers
│       └── test/java/com/enterprise/app/
│           ├── controller/ProductControllerTest.java
│           └── service/ProductServiceTest.java
│
├── jenkins/
│   └── Jenkinsfile                        # 21-stage declarative pipeline
│
├── terraform/
│   ├── environments/
│   │   ├── dev/main.tf                    # Dev environment config
│   │   ├── staging/main.tf
│   │   └── prod/main.tf
│   └── modules/
│       └── eks/                           # Reusable EKS module (VPC, cluster, ECR, IAM)
│
├── ansible/
│   ├── playbooks/
│   │   └── configure-environment.yml      # Pre-deploy configuration playbook
│   ├── roles/
│   │   ├── app-deploy/tasks/main.yml      # K8s ConfigMaps, ServiceMonitors
│   │   ├── docker-setup/
│   │   ├── k8s-setup/
│   │   └── monitoring-setup/tasks/main.yml # Prometheus + Grafana install
│   └── inventories/
│       ├── dev/hosts.yml
│       ├── staging/hosts.yml
│       └── prod/hosts.yml
│
├── k8s/
│   ├── base/values.yaml                   # Default Helm values
│   └── overlays/
│       ├── dev/values.yaml                # Dev overrides (1 replica, no TLS)
│       ├── staging/values.yaml
│       └── prod/values.yaml               # Prod (3 replicas, PDB=2, HPA 3-20)
│
├── sonarqube/
│   └── sonar-project.properties           # SonarQube scanner config + quality gate docs
│
├── nexus/
│   └── setup.groovy                       # Nexus initial configuration script
│
├── prometheus/
│   ├── rules.yaml                         # Alert rules (PrometheusRule CRD)
│   └── scrape-config.yaml                 # Scrape targets for app + tools
│
└── grafana/
    └── dashboards/
        └── enterprise-cicd.json           # Grafana dashboard (request rate, latency, memory, deployments)

Pipeline Flow

Every git push to any branch triggers the pipeline via GitHub webhook. The full flow for a push to main:

git push → GitHub → Jenkins Webhook
  │
  ├─ Stage 1:  Checkout & Version
  │            Compute commit SHA, semantic version, image tag
  │
  ├─ Stage 2:  SAST — SonarQube
  │            Static analysis: bugs, vulnerabilities, code smells, hotspots
  │
  ├─ Stage 3:  Quality Gate
  │            BLOCKS if coverage < 70%, reliability/security rating < A, new vulnerabilities > 0
  │
  ├─ Stage 4:  Unit Tests + Coverage
  │            Maven verify, JaCoCo report, BLOCKS if coverage < 70%
  │            Publishes JUnit XML and HTML coverage report to Jenkins
  │
  ├─ Stage 5:  OWASP Dependency Check
  │            Scans all dependencies for known CVEs (NVD database)
  │            BLOCKS if any dependency has CVSS score >= 7.0
  │
  ├─ Stage 6:  Build JAR
  │            Maven package, archives .jar as Jenkins artifact
  │
  ├─ Stage 7:  Publish to Nexus
  │            Deploys JAR to Nexus maven-releases or maven-snapshots
  │            Only runs on main/develop branches
  │
  ├─ Stage 8:  Docker Build
  │            Multi-stage build: deps cache → build+test → distroless runtime
  │            Tags with commit SHA + build number
  │
  ├─ Stage 9:  Trivy Container Scan
  │            BLOCKS if CRITICAL CVE found in image
  │            Full JSON report archived for audit trail
  │
  ├─ Stage 10: Push to Registry
  │            Authenticates to Nexus Docker registry
  │            Pushes versioned tag + latest (main branch only)
  │
  ├─ Stage 11: Ansible — Configure Dev
  │            Runs configure-environment.yml against dev inventory
  │            Creates/updates ConfigMaps, ServiceMonitors
  │
  ├─ Stage 12: Deploy to Dev
  │            Helm upgrade --atomic (auto-rollback on failure)
  │            kubectl rollout status verify
  │
  ├─ Stage 13: Smoke Tests — Dev
  │            Health endpoint retry loop (12 attempts × 10s)
  │            API endpoint validation
  │
  ├─ Stage 14: Ansible — Configure Staging    [main branch only]
  │
  ├─ Stage 15: Deploy to Staging              [main branch only]
  │
  ├─ Stage 16: Integration Tests              [main branch only]
  │            Maven failsafe plugin, full API test suite
  │
  ├─ Stage 17: Manual Approval Gate           [main branch only]
  │            Slack notification with approve/reject link
  │            24-hour timeout before auto-cancellation
  │
  ├─ Stage 18: Ansible — Configure Prod       [approved main only]
  │
  ├─ Stage 19: Deploy to Production           [approved main only]
  │            Helm upgrade --atomic, zero-downtime rolling update
  │
  ├─ Stage 20: Smoke Tests — Production       [approved main only]
  │            Health + API validation against production endpoint
  │
  └─ Stage 21: Grafana Annotation             [main branch]
               Posts deployment annotation to Grafana
               Visible on dashboards to correlate deploys with metric changes

Tool Deep Dives

GitHub — Source Control

The pipeline uses GitHub webhooks to trigger Jenkins on every push. Configure this once and every subsequent push triggers a build within seconds.

Setting up the webhook:

In your GitHub repo: Settings → Webhooks → Add webhook
Payload URL: http://YOUR_JENKINS_URL/github-webhook/
Content type: application/json
Select: "Just the push event" + "Pull requests"
Active: checked

Branch protection rules (Settings → Branches → Add rule for main):

Require pull request reviews before merging
Require status checks to pass: continuous-integration/jenkins
Require branches to be up to date before merging
Include administrators

Repository secrets (Settings → Secrets and variables → Actions): These are not used by Actions but document what external systems need credentials. The actual credentials are stored in Jenkins Credentials Manager.

Jenkins — Orchestration

Jenkins runs the pipeline. The Jenkinsfile in jenkins/ is loaded automatically when you configure the job to use SCM.

Architecture decision — Kubernetes pod agents:

Instead of permanent Jenkins agents, this pipeline uses Kubernetes pod agents. Each build gets a fresh pod containing exactly the containers it needs (maven, docker, trivy, ansible, kubectl). When the build finishes, the pod is deleted. Benefits:

No stale build environments
Parallel builds scale automatically (Kubernetes scheduler handles it)
No wasted compute when no builds are running
Clean isolation between builds

Required Jenkins plugins:

Plugin	Purpose
Pipeline	Core declarative pipeline support
Kubernetes	Pod-based dynamic agents
SonarQube Scanner	waitForQualityGate + withSonarQubeEnv
JaCoCo	Code coverage reports and enforcement
HTML Publisher	OWASP and JaCoCo HTML reports
Slack Notification	Deployment alerts
Docker Pipeline	Docker build/push within pipeline
Kubernetes CLI	withKubeConfig step
AnsiColor	Colored console output
Build Discarder	Automatic old build cleanup
Timestamper	Build timestamps in console
GitHub	GitHub webhook integration

Required Jenkins Credentials (Manage Jenkins → Credentials → Global):

Credential ID	Type	Description
`nexus-url`	Secret text	Nexus server URL
`nexus-credentials`	Username/password	Nexus deploy user
`sonar-url`	Secret text	SonarQube server URL
`sonar-token`	Secret text	SonarQube auth token
`docker-registry-url`	Secret text	Nexus Docker registry URL
`registry-credentials`	Username/password	Docker registry login
`grafana-url`	Secret text	Grafana server URL
`grafana-api-key`	Secret text	Grafana API key
`kubeconfig-dev`	Secret file	kubeconfig for dev cluster
`kubeconfig-staging`	Secret file	kubeconfig for staging cluster
`kubeconfig-prod`	Secret file	kubeconfig for prod cluster
`slack-token`	Secret text	Slack bot OAuth token

Creating the pipeline job:

New Item → Pipeline
Name: enterprise-cicd-platform
Pipeline → Definition: Pipeline script from SCM
SCM: Git → Repository URL: your GitHub repo
Branch: */main
Script Path: jenkins/Jenkinsfile
Save → Build Now

Global environment variables (Manage Jenkins → System → Global Properties → Environment Variables):

AWS_REGION = us-east-1
AWS_ACCOUNT_ID = 123456789012

Maven — Build Tool

Maven handles compilation, testing, dependency management, and artifact publishing. The pom.xml in app/ includes:

Key Maven plugins:

Plugin	Purpose	Configured Behavior
spring-boot-maven-plugin	Build executable JAR + build-info	Produces layered JAR for Docker optimization
jacoco-maven-plugin	Code coverage	Fails if line coverage < 70%, branch < 60%
dependency-check-maven	OWASP CVE scan	Fails if CVSS >= 7.0
maven-surefire-plugin	Unit test runner	Produces JUnit XML for Jenkins

Dependency management through Nexus:

The pom.xml defines <repositories> pointing to Nexus instead of Maven Central. This means:

mvn install fetches from http://nexus:8081/repository/maven-public/
Nexus proxies Maven Central and caches everything locally
Builds work even when Maven Central is unreachable
Full audit trail of every dependency used

Local Maven build:

cd app

# Full build with tests and coverage
mvn clean verify

# Skip tests for faster local iteration
mvn package -DskipTests

# Run SonarQube analysis
mvn sonar:sonar -Dsonar.host.url=http://localhost:9000 -Dsonar.login=YOUR_TOKEN

# Deploy artifact to Nexus
mvn deploy -Dusername=admin -Dpassword=admin123 -DNEXUS_URL=http://localhost:8081

Maven settings.xml (configured as a Kubernetes ConfigMap, mounted into maven container):

<settings>
  <servers>
    <server>
      <id>nexus-releases</id>
      <username>${env.NEXUS_USER}</username>
      <password>${env.NEXUS_PASSWORD}</password>
    </server>
    <server>
      <id>nexus-snapshots</id>
      <username>${env.NEXUS_USER}</username>
      <password>${env.NEXUS_PASSWORD}</password>
    </server>
  </servers>
  <mirrors>
    <mirror>
      <id>nexus</id>
      <mirrorOf>*</mirrorOf>
      <url>http://nexus:8081/repository/maven-public/</url>
    </mirror>
  </mirrors>
</settings>

SonarQube — Code Quality

SonarQube performs static analysis on the Java source code. The pipeline has a hard Quality Gate — if the gate fails, the pipeline stops and no artifact is built.

Quality Gate configuration (Enterprise Gate):

Apply these conditions in SonarQube → Quality Gates → Create → Add Condition:

Metric	Operator	Value	Effect
Coverage	Less than	70%	Fail
Duplicated Lines (%)	Greater than	10%	Fail
Reliability Rating	Worse than	A	Fail
Security Rating	Worse than	A	Fail
Maintainability Rating	Worse than	A	Fail
Security Hotspots Reviewed	Less than	100%	Fail
New Bugs	Greater than	0	Fail
New Vulnerabilities	Greater than	0	Fail

Set this as the default gate for the project: SonarQube → Projects → enterprise-cicd-app → Project Settings → Quality Gate → Enterprise Gate

SonarQube in Jenkins:

Install SonarQube Scanner plugin
Manage Jenkins → Configure System → SonarQube servers:
- Name: SonarQube (must match withSonarQubeEnv('SonarQube') in Jenkinsfile)
- URL: http://sonarqube:9000
- Authentication token: create in SonarQube → My Account → Security → Generate Token

SonarQube → Nexus integration:

SonarQube can also be configured to publish analysis results. More importantly, if SonarQube marks a project as failing, the Jenkins waitForQualityGate step will fail the entire pipeline before the JAR is even uploaded to Nexus. This means Nexus never receives artifacts that fail quality standards.

Nexus — Artifact Repository

Nexus Repository Manager stores and proxies artifacts. This pipeline uses Nexus for both Maven JARs and Docker images.

Repository configuration (see nexus/setup.groovy):

Repository	Type	Purpose
`maven-releases`	hosted	Stores production JARs (ALLOW_ONCE policy)
`maven-snapshots`	hosted	Stores SNAPSHOT builds
`maven-central`	proxy	Caches Maven Central
`maven-public`	group	Single URL for all Maven repos
`docker-hosted`	hosted	Internal Docker registry (port 8082)
`docker-hub`	proxy	Caches Docker Hub pulls
`docker-group`	group	Single registry URL (port 8083)

Apply Nexus initial configuration:

# First, get the admin password
docker exec nexus cat /nexus-data/admin.password

# Apply setup script via REST API
curl -u admin:INITIAL_PASSWORD \
  -X POST "http://localhost:8081/service/rest/v1/script" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"setup\", \"type\": \"groovy\", \"content\": \"$(cat nexus/setup.groovy | jq -Rs .)\"}"

curl -u admin:INITIAL_PASSWORD \
  -X POST "http://localhost:8081/service/rest/v1/script/setup/run"

Configure Docker registry in Nexus:

Security → Realms → Add "Docker Bearer Token Realm" to Active
This enables docker login against Nexus

Browsing stored artifacts:

# List Maven releases
curl -u admin:admin123 \
  "http://localhost:8081/service/rest/v1/components?repository=maven-releases"

# List Docker images
curl -u admin:admin123 \
  "http://localhost:8081/service/rest/v1/components?repository=docker-hosted"

Docker — Containerization

The docker/Dockerfile uses a three-stage build:

Stage 1 — Dependency Cache: Copies only pom.xml and runs dependency:go-offline. Because dependencies change less frequently than code, this layer is cached separately. A code change only invalidates stage 2 and 3, not the dependency download.

Stage 2 — Build: Copies source code and runs mvn clean verify. Then uses Spring Boot's layertools to extract the JAR into separate directories: dependencies → spring-boot-loader → snapshot-dependencies → application. Each layer goes into the Docker image separately, so a code-only change (no dependency change) only rebuilds the application layer.

Stage 3 — Runtime (Distroless): Uses gcr.io/distroless/java21-debian12:nonroot. This base image contains:

Java 21 JRE
Minimal system libraries
NO shell (/bin/sh, /bin/bash)
NO package manager (apt, apk)
Runs as UID 65532 (nonroot variant)

The attack surface is minimal. An attacker who exploits a vulnerability in the application cannot exec into the container to run commands — there is no shell to exec into.

JVM flags in the ENTRYPOINT:

-XX:+UseContainerSupport     Respect cgroup CPU and memory limits
-XX:MaxRAMPercentage=75.0    Use 75% of container memory for heap
-XX:+ExitOnOutOfMemoryError  Crash fast, let K8s restart cleanly
-Djava.security.egd=...      Fast entropy for SecureRandom

Building the image locally:

docker build \
  --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
  --build-arg VERSION=1.0.0-local \
  -f docker/Dockerfile \
  -t enterprise-cicd-app:local \
  .

# Scan for CVEs before running
trivy image enterprise-cicd-app:local --severity HIGH,CRITICAL

# Run locally
docker run -p 8080:8080 enterprise-cicd-app:local
curl http://localhost:8080/actuator/health

Kubernetes — Orchestration

The application deploys to Kubernetes via Helm. Three environments — dev, staging, prod — are managed with a base values.yaml and environment-specific overrides.

Helm chart structure (k8s/):

The chart provisions:

Deployment — rolling update strategy, zero maxUnavailable replicas
Service — ClusterIP with Prometheus annotations
Ingress — nginx with SSL redirect and rate limiting
HorizontalPodAutoscaler — CPU and memory based scaling
PodDisruptionBudget — availability during node drains
ServiceAccount — with IRSA annotation for AWS permissions

Environment differences:

Setting	Dev	Staging	Prod
Replicas	1	2	3
HPA	Disabled	2–10	3–20
PDB minAvailable	Disabled	1	2
TLS	No	Yes	Yes
Resources	Small	Medium	Large

Manual Helm operations:

# Deploy to dev
helm upgrade --install enterprise-cicd-app ./k8s/helm/enterprise-cicd-app \
  --namespace enterprise-cicd-app-dev --create-namespace \
  --values k8s/base/values.yaml \
  --values k8s/overlays/dev/values.yaml \
  --set image.tag=abc1234-42 \
  --atomic --timeout 5m

# Check rollout status
kubectl rollout status deployment/enterprise-cicd-app -n enterprise-cicd-app-prod

# View pods across AZs
kubectl get pods -n enterprise-cicd-app-prod -o wide

# Check HPA status
kubectl get hpa -n enterprise-cicd-app-prod

# Rollback
helm rollback enterprise-cicd-app 0 -n enterprise-cicd-app-prod

Terraform — Infrastructure as Code

Terraform provisions the AWS infrastructure. The dev, staging, and prod environments each have their own state file in S3 with DynamoDB locking.

What Terraform provisions:

VPC with public/private subnets across 3 AZs
NAT Gateway (single in dev, multi in prod for HA)
EKS cluster (v1.29) with managed node groups
Application node group (t3.medium in dev, larger in prod)
Monitoring node group (isolated for Prometheus/Grafana)
ECR repository with immutable tags, auto lifecycle policy
KMS keys for EKS secrets, EBS volumes
IAM roles with IRSA for Jenkins agent, app workloads, EBS CSI driver
Security groups (bastion, EKS, ALB)

Terraform workflow:

cd terraform/environments/dev

# Initialize (downloads providers, configures S3 backend)
terraform init

# Preview changes
terraform plan -out=tfplan

# Apply
terraform apply tfplan

# Get cluster credentials
aws eks update-kubeconfig \
  --name enterprise-cicd-dev \
  --region us-east-1

# Verify
kubectl cluster-info
kubectl get nodes

State management:

The S3 backend with DynamoDB locking prevents two engineers from running terraform apply simultaneously, which could corrupt state or cause race conditions. Always:

Run terraform plan to see what will change
Share the plan with your team for review on significant changes
Never run apply directly in production without a plan review

Terraform best practices followed in this project:

All resources tagged with Project, Environment, ManagedBy
Remote state in S3 with encryption and versioning
Module abstraction — the EKS module is reused across dev/staging/prod
KMS encryption for EKS secrets and EBS volumes
IRSA instead of node-level IAM roles (pod-level AWS permissions)
single_nat_gateway = var.environment != "prod" — cost optimization in dev, HA in prod

Ansible — Configuration Management

Ansible bridges the gap between infrastructure (Terraform) and application deployment (Helm). It runs in the Jenkins pipeline before each Helm deploy.

Why Ansible alongside Terraform and Helm?

Terraform manages infrastructure lifecycle (create, update, destroy). Helm manages Kubernetes application deployment. Ansible handles the middle layer: things that happen on the running infrastructure but aren't part of the application deployment:

Verifying cluster connectivity
Creating/updating Kubernetes ConfigMaps with environment-specific config
Applying Prometheus ServiceMonitors (so the app is scraped immediately after deploy)
Installing cluster-wide tooling (Prometheus stack via Helm, managed by Ansible)
Running pre-flight checks

Key Ansible patterns used:

Roles — reusable units of automation. The app-deploy role handles K8s resource management. The monitoring-setup role handles Prometheus/Grafana installation. Roles can be called from multiple playbooks.

Inventories — separate inventory files for dev, staging, prod. Each points to the correct nodes with the right SSH keys and environment variables.

Vault — secrets (passwords, API keys) are encrypted with ansible-vault. The Jenkins pipeline mounts the vault password from a Kubernetes Secret.

Running Ansible manually:

cd ansible

# Run full pre-deploy configuration for dev
ansible-playbook playbooks/configure-environment.yml \
  -i inventories/dev/hosts.yml \
  --extra-vars "app_name=enterprise-cicd-app app_version=1.0.0 image_tag=abc1234-42 env=dev" \
  --ask-vault-pass \
  -v

# Dry run (check mode)
ansible-playbook playbooks/configure-environment.yml \
  -i inventories/dev/hosts.yml \
  --extra-vars "app_name=enterprise-cicd-app app_version=1.0.0 image_tag=abc1234-42 env=dev" \
  --check --diff

# Run specific role only
ansible-playbook playbooks/configure-environment.yml \
  -i inventories/dev/hosts.yml \
  --tags "app-deploy"

Prometheus — Metrics Collection

Prometheus scrapes metrics from every component in the platform. In Kubernetes, it is installed as part of the kube-prometheus-stack Helm chart (managed by Ansible's monitoring-setup role).

What Prometheus scrapes:

Target	Metrics	Scrape Interval
Spring Boot app (all envs)	HTTP request rate, latency, JVM heap, custom business metrics	30s (15s prod)
Kubernetes nodes	CPU, memory, disk, network	60s
Kubernetes pods	Container resource usage, restart counts	30s
Jenkins	Build duration, queue length, failed builds	60s
SonarQube	Code quality metrics	60s
Nexus	Repository stats, JVM, request rates	60s

Spring Boot metrics exposed (/actuator/prometheus):

Metric	Type	Description
`http_server_requests_seconds`	Histogram	HTTP request latency by URI, method, status
`jvm_memory_used_bytes`	Gauge	JVM heap and non-heap usage
`jvm_gc_pause_seconds`	Summary	GC pause latency
`process_cpu_usage`	Gauge	CPU percentage
`app_products_created_total`	Counter	Custom business metric
`app_products_deleted_total`	Counter	Custom business metric
`app_products_count`	Gauge	Current product count
`app_products_findall_duration_seconds`	Timer	Service method latency

Alert rules (see prometheus/rules.yaml):

Alert	Condition	Severity
AppDown	Instance not responding	Critical
AppHighErrorRate	HTTP 5xx > 5%	High
AppHighLatency	p95 > 2 seconds	High
AppHighMemoryUsage	Heap > 85%	Medium
PodCrashLooping	> 3 restarts in 15m	High
DeploymentReplicasMismatch	Available != Desired	High
HPAMaxReplicasReached	HPA at max for 15m	Medium

Applying alert rules:

# Apply PrometheusRule CRD
kubectl apply -f prometheus/rules.yaml

# Verify rules are loaded
kubectl get prometheusrules -n monitoring
curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].name'

Grafana — Observability Dashboard

Grafana visualizes Prometheus metrics. The dashboard in grafana/dashboards/enterprise-cicd.json includes:

Dashboard panels:

Panel	Type	Shows
Request Rate	Stat	Current req/s in production
Error Rate	Stat	HTTP 5xx percentage — green/yellow/red thresholds
P95 Latency	Stat	95th percentile latency
Active Pods	Stat	Current replica count
HTTP Rate by Endpoint	Time series	Per-endpoint request rates over time
JVM Heap Memory	Time series	Used vs max heap over time
Products Created	Time series	Custom business metric rate
Pod Restart Count	Time series	Restart events per pod
Deployment Annotations	Annotation list	Last 10 deployments

Deployment annotations — the most powerful feature: Every production deployment, Jenkins calls the Grafana Annotations API. A vertical line appears on all time series charts at the exact time of the deployment. When you see a spike in error rate, you can instantly see "did a deployment just happen?" without checking Jenkins.

Importing the dashboard:

Grafana → Dashboards → Import
Upload grafana/dashboards/enterprise-cicd.json
Select Prometheus as the data source
Click Import

Provisioning dashboards automatically (via Ansible): The monitoring-setup role creates a ConfigMap from the JSON file and mounts it into Grafana via the dashboard provider configuration. Dashboards are available immediately on fresh installs.

Application

The Spring Boot application is a Products REST API that demonstrates production patterns: structured logging, Prometheus instrumentation, input validation, custom health indicators, and Spring Security with security response headers.

Endpoints:

# Products API
GET    /api/v1/products          # List all products
GET    /api/v1/products/{id}     # Get by ID (404 if not found)
POST   /api/v1/products          # Create (validates input, 400 on invalid)
PUT    /api/v1/products/{id}     # Update
DELETE /api/v1/products/{id}     # Delete
GET    /api/v1/products/count    # Count — smoke test target

# Spring Boot Actuator
GET    /actuator/health          # Full health check
GET    /actuator/health/liveness  # Kubernetes liveness probe
GET    /actuator/health/readiness # Kubernetes readiness probe
GET    /actuator/prometheus       # Prometheus metrics (3000+ metrics)
GET    /actuator/info             # Build version + environment info

Run locally:

cd app

# Run with Maven
mvn spring-boot:run

# Or build and run the JAR
mvn package -DskipTests
java -jar target/enterprise-cicd-app-*.jar

# Access the API
curl http://localhost:8080/api/v1/products | jq
curl http://localhost:8080/actuator/health | jq
curl -X POST http://localhost:8080/api/v1/products \
  -H "Content-Type: application/json" \
  -d '{"name":"Test","description":"A product","price":19.99,"category":"test"}'

Security Controls

Layer	Control	Tool	Scope
Code	SAST	SonarQube	Every commit
Code	Quality gate enforcement	SonarQube	Every commit
Dependencies	CVE scan	OWASP Dependency Check	Every build
Container	CVE scan	Trivy	Every Docker build
Container	Multi-stage distroless build	Docker	Build time
Container	Non-root runtime (UID 65532)	Dockerfile	Runtime
Container	No shell in final image	Distroless	Runtime
API	Security headers	Spring Security	Runtime
K8s	Non-root pod	securityContext	Deployment
K8s	Read-only root filesystem	securityContext	Deployment
K8s	Drop all capabilities	securityContext	Deployment
K8s	Pod-level AWS permissions	IRSA	Deployment
Infra	KMS encryption for secrets	Terraform	Infrastructure
Infra	Encrypted EBS volumes	Terraform	Infrastructure
Infra	Private cluster endpoint (prod)	Terraform	Infrastructure
Secrets	Ansible Vault	Ansible	Configuration

Prerequisites

Local machine:

Java 21 (java -version)
Maven 3.9+ (mvn -version)
Docker 24+ (docker --version)
kubectl 1.29+ (kubectl version --client)
Helm 3.14+ (helm version)
Terraform 1.7+ (terraform version)
Ansible 2.16+ (ansible --version)
AWS CLI v2 (aws --version)

AWS:

AWS account with permissions to create EKS, VPC, ECR, IAM, KMS
S3 bucket for Terraform state: enterprise-cicd-tfstate
DynamoDB table for Terraform lock: enterprise-cicd-tflock

Services (all included in docker-compose.yml):

Jenkins, SonarQube, Nexus, Prometheus, Grafana

Local Development Setup

# 1. Clone the repository
git clone https://github.com/YOUR_USERNAME/enterprise-cicd-platform.git
cd enterprise-cicd-platform

# 2. Start the tool stack
docker compose -f docker/docker-compose.yml up -d

# Wait for services to start (~3-5 minutes)
docker compose -f docker/docker-compose.yml ps

# 3. Access the services
open http://localhost:8080   # Jenkins (setup wizard on first run)
open http://localhost:9000   # SonarQube (admin/admin — change on first login)
open http://localhost:8081   # Nexus (admin/admin123 — change on first login)
open http://localhost:9090   # Prometheus
open http://localhost:3000   # Grafana (admin/admin)

# 4. Apply Nexus configuration
docker exec nexus cat /nexus-data/admin.password  # Get initial password
# Then configure repositories via Nexus UI or the setup.groovy script

# 5. Build the application
cd app
mvn clean verify

# 6. Build Docker image
cd ..
docker build -f docker/Dockerfile -t enterprise-cicd-app:local .

# 7. Run the application locally
docker run -p 8080:8080 enterprise-cicd-app:local
curl http://localhost:8080/actuator/health

Jenkins Setup Guide

# 1. Get initial admin password
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

# 2. Open http://localhost:8080 and complete setup wizard
# Select: Install suggested plugins + the additional plugins listed above

# 3. Configure SonarQube server
# Manage Jenkins → Configure System → SonarQube servers
# Name: SonarQube  URL: http://sonarqube:9000

# 4. Configure Maven settings.xml as a ConfigMap (if using K8s agents)
kubectl create configmap maven-settings-cm \
  --from-file=settings.xml=ansible/roles/app-deploy/files/settings.xml \
  -n jenkins

# 5. Create pipeline job
# New Item → Pipeline → Configure SCM → Jenkinsfile path: jenkins/Jenkinsfile

# 6. Add all credentials (see Jenkins Setup Guide section above)

# 7. Trigger first build
# Build Now → watch pipeline execute each stage

Infrastructure Provisioning

# 1. Create Terraform state backend (one-time)
aws s3 mb s3://enterprise-cicd-tfstate --region us-east-1
aws s3api put-bucket-versioning \
  --bucket enterprise-cicd-tfstate \
  --versioning-configuration Status=Enabled

aws dynamodb create-table \
  --table-name enterprise-cicd-tflock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

# 2. Deploy dev infrastructure
cd terraform/environments/dev
terraform init
terraform plan -out=tfplan
terraform apply tfplan

# 3. Configure kubectl
aws eks update-kubeconfig \
  --name enterprise-cicd-dev \
  --region us-east-1

kubectl get nodes

# 4. Install monitoring (via Ansible)
cd ansible
ansible-playbook playbooks/configure-environment.yml \
  -i inventories/dev/hosts.yml \
  --extra-vars "app_name=enterprise-cicd-app app_version=init env=dev image_tag=init" \
  --tags "monitoring-setup"

Monitoring Setup

# Verify Prometheus is running
kubectl get pods -n monitoring

# Port-forward Prometheus to localhost
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring &
open http://localhost:9090

# Port-forward Grafana to localhost
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring &
open http://localhost:3000

# Import the application dashboard
# Grafana → Dashboards → Import → Upload grafana/dashboards/enterprise-cicd.json

# Verify app is being scraped
curl -s "http://localhost:9090/api/v1/targets" | \
  jq '.data.activeTargets[] | select(.labels.job | contains("enterprise-cicd")) | {job: .labels.job, health: .health}'

# Test an alert is firing
# (simulate by scaling deployment to 0)
kubectl scale deployment enterprise-cicd-app --replicas=0 -n enterprise-cicd-app-prod
# Wait 1 minute, then check Prometheus alerts
curl -s "http://localhost:9090/api/v1/alerts" | jq '.data.alerts[] | {name: .labels.alertname, state: .state}'
# Scale back up
kubectl scale deployment enterprise-cicd-app --replicas=3 -n enterprise-cicd-app-prod

Runbook — Common Operations

Triggering a manual build

# Via Jenkins CLI
java -jar jenkins-cli.jar -s http://jenkins:8080 build enterprise-cicd-platform -p BRANCH=main

# Via GitHub: push any commit to trigger webhook
git commit --allow-empty -m "chore: trigger pipeline"
git push

Rolling back a production deployment

# Option 1: Helm rollback (recommended)
helm rollback enterprise-cicd-app 0 -n enterprise-cicd-app-prod

# Option 2: kubectl rollout undo
kubectl rollout undo deployment/enterprise-cicd-app -n enterprise-cicd-app-prod

# Option 3: Re-run Jenkins pipeline with previous image tag
# Jenkins → Build with Parameters → set IMAGE_TAG to previous value

Checking why a pod is failing

kubectl describe pod <pod-name> -n enterprise-cicd-app-prod
kubectl logs <pod-name> -n enterprise-cicd-app-prod --previous
kubectl get events -n enterprise-cicd-app-prod --sort-by='.lastTimestamp'

Checking SonarQube quality gate status

curl -u admin:admin \
  "http://localhost:9000/api/qualitygates/project_status?projectKey=enterprise-cicd-app" | jq

Finding a JAR version in Nexus

curl -u admin:admin123 \
  "http://localhost:8081/service/rest/v1/search?repository=maven-releases&group=com.enterprise&name=enterprise-cicd-app" | jq

Scaling the application

# Manual scale (HPA will revert this after a few minutes)
kubectl scale deployment enterprise-cicd-app --replicas=8 -n enterprise-cicd-app-prod

# Update HPA min/max via Helm
helm upgrade enterprise-cicd-app ./k8s/helm/enterprise-cicd-app \
  -n enterprise-cicd-app-prod \
  --set autoscaling.maxReplicas=25 \
  --reuse-values

Troubleshooting

Jenkins pipeline fails at "Publish to Nexus":

Verify nexus-url and nexus-credentials credentials exist in Jenkins
Check Nexus is reachable: curl -sf http://nexus:8081/service/rest/v1/status
Verify maven-releases repository exists in Nexus
Check ALLOW_ONCE policy is not rejecting a re-publish of same version

SonarQube quality gate always passes even with bad code:

Verify the project is using "Enterprise Gate" not the default "Sonar way"
SonarQube → Projects → enterprise-cicd-app → Project Settings → Quality Gate

Helm deploy fails with "atomic" flag:

This means the deployment rolled back automatically (--atomic behavior)
Check: kubectl describe pod -n enterprise-cicd-app-dev -l app=enterprise-cicd-app
Common causes: image not found in registry, resource limits too low, health probe failing

Trivy scan fails on distroless image with false positives:

Add to .trivyignore in repo root: CVE-XXXX-XXXX one per line
Or update the image to a newer distroless tag: gcr.io/distroless/java21-debian12:nonroot

Ansible playbook fails with "kubectl: command not found":

The Ansible container in the Jenkins pod doesn't have kubectl installed by default
Either: add kubectl to the Ansible image, or use the kubernetes.core collection which doesn't require kubectl

Branch Strategy

Branch	Purpose	Pipeline Stages	Deploys to
`feature/*`	New features	Stages 1-10 only (no deploy)	Nothing
`develop`	Integration branch	All stages through dev smoke tests	Dev
`main`	Production-ready code	All 21 stages	Dev → Staging → Prod (with approval)
`hotfix/*`	Emergency fixes	All stages, bypass staging	Dev → Prod (with approval)

Git workflow:

Create feature branch from develop
Open PR → Jenkins runs CI stages (build, test, sonar, trivy)
PR review + merge to develop
develop pipeline deploys to dev
When ready for release: PR from develop to main
main pipeline runs full flow, waits for approval before production

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/add-secret-scanning)
Make changes — ensure tests pass locally (mvn clean verify)
Run SonarQube locally to check for quality issues
Commit with a conventional commit message (feat:, fix:, chore:)
Open a pull request — Jenkins CI will run automatically
Address any quality gate failures or reviewer feedback
Merge after approval

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
ansible		ansible
app		app
docker		docker
grafana/dashboards		grafana/dashboards
helm/enterprise-app		helm/enterprise-app
jenkins		jenkins
k8s		k8s
monitoring		monitoring
nexus		nexus
prometheus		prometheus
sonarqube		sonarqube
terraform		terraform
.gitignore		.gitignore
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation