[ALERT] cmp-dev-alb-target-response-time

[alexeygrigorev]

This issue was opened automatically from a CloudWatch alarm notification.

Alarm

Field	Value
Name	cmp-dev-alb-target-response-time
State	ALARM
Changed at	2026-08-25T07:57:59.875+0000
Description	ALB target p95 response time is high for CMP dev.
Reason	Threshold Crossed: 2 datapoints [5.107049986271482 (25/08/26 07:50:00), 5.071076734574111 (25/08/26 07:45:00)] were greater than the threshold (5.0).
AWS account	387546586013
Region	eu-west-1

Open this alarm in CloudWatch

Metric

Field	Value
Namespace	AWS/ApplicationELB
Metric	TargetResponseTime
Dimensions	TargetGroup=targetgroup/course-management-dev/e7fbcd61f2279f3a, LoadBalancer=app/course-management-shared/bd279e4cc4f97601
Statistic	p95
Period	300 seconds
Evaluation periods	2
Threshold	GreaterThanThreshold 5.0

Investigation

• Check application and load-balancer logs around the state-change time.
• Check the current service version at /api/health/.
• Record the cause and resolution here before closing the issue.

Application logs

Field	Value
Log group	/ecs/course-management-dev
Suggested window	2026-08-25T07:47:59.875000+00:00 to 2026-08-25T08:02:59.875000+00:00
Investigator role	arn:aws:iam::387546586013:role/cmp-alert-investigator

Open CloudWatch Logs Insights, select /ecs/course-management-dev, or use the
cmp-alert-investigator AWS CLI profile described in the infrastructure README:

aws --profile cmp-alert-investigator logs filter-log-events \
  --region eu-west-1 \
  --log-group-name /ecs/course-management-dev \
  --start-time 1787644079875 \
  --end-time 1787644979875 \
  --limit 100

Review log output for sensitive information before copying any excerpt into
this public issue.

Raw CloudWatch notification

{
  "AWSAccountId": "387546586013",
  "ActionExecutedAfterMuteWindow": false,
  "AlarmActions": [
    "arn:aws:sns:eu-west-1:387546586013:course-management-operator-alerts"
  ],
  "AlarmArn": "arn:aws:cloudwatch:eu-west-1:387546586013:alarm:cmp-dev-alb-target-response-time",
  "AlarmConfigurationUpdatedTimestamp": "2026-07-08T22:06:44.841+0000",
  "AlarmDescription": "ALB target p95 response time is high for CMP dev.",
  "AlarmName": "cmp-dev-alb-target-response-time",
  "InsufficientDataActions": [],
  "NewStateReason": "Threshold Crossed: 2 datapoints [5.107049986271482 (25/08/26 07:50:00), 5.071076734574111 (25/08/26 07:45:00)] were greater than the threshold (5.0).",
  "NewStateValue": "ALARM",
  "OKActions": [],
  "OldStateValue": "OK",
  "Region": "EU (Ireland)",
  "StateChangeTime": "2026-08-25T07:57:59.875+0000",
  "Trigger": {
    "ComparisonOperator": "GreaterThanThreshold",
    "Dimensions": [
      {
        "name": "TargetGroup",
        "value": "targetgroup/course-management-dev/e7fbcd61f2279f3a"
      },
      {
        "name": "LoadBalancer",
        "value": "app/course-management-shared/bd279e4cc4f97601"
      }
    ],
    "EvaluateLowSampleCountPercentile": "",
    "EvaluationPeriods": 2,
    "ExtendedStatistic": "p95",
    "MetricName": "TargetResponseTime",
    "Namespace": "AWS/ApplicationELB",
    "Period": 300,
    "StatisticType": "ExtendedStatistic",
    "Threshold": 5.0,
    "TreatMissingData": "notBreaching",
    "Unit": null
  }
}

[alexeygrigorev]

Root cause

Not an application bug — a deploy-cadence artifact. Four small commits (country-list additions) landed on main within ~15 minutes on 2026-08-25 (07:27:53–07:41:18 UTC), each triggering its own deploy-dev.yaml run. course-management-dev runs a single ECS task (desired_count = 1), so each deploy cycles that one task through a cold start. ECS service events show three task replacements in under 8 minutes (new tasks started at 07:45:39, 07:47:24, 07:50:11 UTC), overlapping exactly with this alarm's breaching p95 datapoints (07:45–07:55 UTC). No application errors correlate with the window — it's cold-start/connection-drain latency from stacked deploys on a single-instance environment.

Fixes

1. Commit 5a3a95b: added a concurrency guard to deploy-dev.yaml so a new push cancels an in-progress dev deploy instead of stacking on top of it — only the latest commit's deploy reaches ECS. Also serialized deploy-prod.yaml dispatches. Deployed to dev and verified (/api/health/ reports 20260826-082528-5a3a95b).
2. Infra: dev is expected to have brief deploy-time blips by design (it redeploys on every push), so alerting on it isn't actionable — only prod should page. Opened DataTalksClub/aws-infra#27 to remove all cmp-dev-* CloudWatch alarms (including this one) and keep alerting scoped to prod. Not yet merged/applied.

Leaving this open until the infra PR is merged and applied.