v0.6.1

Fixed

• Glue RCA false-positive failure classification: verifyGlueRCA Check 2 queried /aws-glue/jobs/error without a FilterPattern, causing benign stderr messages (e.g. "Preparing ...") to be misclassified as failures. Added FilterPattern with error indicators (?Exception ?Error ?FATAL ?Traceback ?OutOfMemoryError ?StackOverflowError) so only genuine errors trigger failure classification. This prevented unnecessary re-runs that doubled Glue compute.
• Watchdog date-boundary test flake: TerminalTriggerRetainsRecord E2E test used sensor data without a date field, causing ResolveExecutionDate to fall back to time.Now(). When the calendar date advanced past the hardcoded trigger date, reconciliation resolved to a different date and fired a spurious TRIGGER_RECOVERED. Sensor data now includes an explicit date.

v0.6.0

STAMP Safety Hardening

Closes 6 STAMP behavioral specification violations with bounded execution, failure classification, and config validation.

Added

• SFN global timeout: TimeoutSeconds on Step Function execution (default 4h, configurable via sfn_timeout_seconds Terraform variable)
• Bounded job poll window: jobPollWindowSeconds pipeline config field (default 1h). Publishes JOB_POLL_EXHAUSTED when exceeded.
• Per-source rerun limits: maxDriftReruns, maxManualReruns, maxCodeRetries with *int pointer semantics (nil = default, 0 = disabled)
• Failure classification: FailureCategory propagated from trigger runner through joblog. PERMANENT failures use maxCodeRetries budget, TRANSIENT/empty use maxRetries
• Pipeline config validation: ValidatePipelineConfig enforces bounds on all retry/rerun fields at config load time (fail-open)
• Configurable trigger retry: trigger_max_attempts Terraform variable (default 3)
• Trigger terminal lifecycle: CompleteTrigger ASL state sets COMPLETED/FAILED_FINAL on trigger records
• Dynamic trigger lock TTL: reads SFN_TIMEOUT_SECONDS env var + 30m buffer

Fixed

• Glue false-success detection via CloudWatch RCA log cross-check
• SLA alert suppression for completed/failed pipelines
• Joblog fallback in SLA guards for TTL-expired or missing triggers
• Watchdog forward-only alerting prevents retroactive SCHEDULE_MISSED on deploy
• Watchdog reconcile mass-triggering guard via joblog terminal check + TTL removal on terminal triggers
• Proactive SLA scheduling skips sensor-triggered pipelines

Removed

• Dead FailureEvaluatorCrash constant

v0.5.2

Hourly SLA deadline offset fix

Offset hourly SLA deadlines to the processing window (H+1) so breach times fall after data actually exists.

Fixed

• Hourly SLA deadline offset: relative deadline :MM for hourly pipelines (date format 2026-03-05T12) now resolves to H+1:MM instead of H:MM. Data for hour H isn't generated until ~H+1:00, so the previous calculation set the breach deadline before data existed — guaranteeing a false breach every execution. Daily pipelines are unchanged.

v0.5.1

Proactive SLA Monitoring

Decouple SLA alerting from Step Function execution so warnings and breaches fire even when pipelines never trigger (data never arrives, sensor fails, trigger missed).

Added

• Proactive SLA monitoring: watchdog Lambda creates EventBridge Scheduler entries for all pipelines with SLA configs. Idempotent via deterministic scheduler names — ConflictException means the schedule already exists and is skipped. (#45)

Changed

• SLA scheduling moved from SFN to watchdog: removed CheckSLAConfig and ScheduleSLAAlerts states from the Step Function (18 → 16 states). The SFN retains only CancelSLASchedules to clean up unfired timers on job completion. (#45)
• CancelSLASchedules accepts deadline/expectedDuration: instead of pre-computed warningAt/breachAt, the cancel handler now receives raw SLA config and recalculates internally. This decouples cancel from the removed scheduling state. (#45)
• Watchdog Lambda now requires SLA_MONITOR_ARN, SCHEDULER_ROLE_ARN, and SCHEDULER_GROUP_NAME environment variables and scheduler:CreateSchedule + iam:PassRole IAM permissions. (#45)

v0.5.0

Added

• Centralized observability pipeline: new event-sink Lambda writes all 14 EventBridge event types to a DynamoDB events table with a GSI for querying by event type and timestamp. New alert-dispatcher Lambda reads from an SQS alert queue and delivers formatted Slack notifications. EventBridge rules route events to both targets automatically. (#41)
• Slack Bot API with message threading: alert-dispatcher uses chat.postMessage with Bot token authentication. Thread records (THREAD#{scheduleId}#{date}) stored in the events table group related alerts into Slack threads by pipeline, schedule, and date. First alert for a pipeline-day creates a new thread; subsequent alerts reply in-thread. (#41)
• SLA warning suppression: fire-alert mode checks BreachAt timestamp before publishing SLA_WARNING. If breach time has already passed, the warning is suppressed to prevent duplicate warning+breach notifications. (#41)
• Manual rerun system: external processes write RERUN_REQUEST# records to the control table. Stream-router validates requests with a circuit breaker that compares sensor updatedAt vs joblog completedAt — rejects reruns when no new data has arrived. (#40)
• Late data arrival detection: stream-router detects sensor updates after job completion and publishes LATE_DATA_ARRIVAL events to EventBridge. (#40)
• Prefix-match sensor keys: stream-router matches sensor keys by prefix for per-period pipelines, enabling a single trigger condition to match sensors keyed with date+hour suffixes. (#38)
• Watchdog trigger reconciliation: watchdog re-evaluates sensor trigger conditions every 5 minutes. If a sensor meets the trigger threshold but no trigger lock exists, the watchdog acquires the lock, starts the Step Function, and publishes a TRIGGER_RECOVERED event. (#44)
• New event types: LATE_DATA_ARRIVAL, RERUN_REJECTED, RETRY_EXHAUSTED, TRIGGER_RECOVERED
• New DynamoDB events table with GSI1 (eventType → timestamp)
• SQS alert queue with dead-letter queue for reliable Slack delivery

Changed

• alert-dispatcher requires SLACK_BOT_TOKEN and SLACK_CHANNEL_ID environment variables (replaces SLACK_WEBHOOK_URL)
• Terraform module exposes slack_bot_token (sensitive) and slack_channel_id variables instead of slack_webhook_url
• Lambda function count: 4 → 6 (added event-sink and alert-dispatcher)

Fixed

• Orchestrator remaps per-period sensor keys during evaluation, preventing key mismatch when sensors use date+hour suffixes (#39)
• ValidationExhausted now ends the Step Functions execution as FAILED instead of SUCCEEDED (#41)
• Joblog entry written on validation exhaustion for audit trail completeness (#41)
• Watchdog prefix-matches trigger records for per-hour pipelines instead of requiring exact key match (#41)
• SLA deadline calculation for daily pipelines with next-day deadlines (e.g., "02:00") now rolls forward 24 hours when the computed breach time is already past (#43)

v0.4.0

Added

• SLA monitoring via EventBridge Scheduler: one-time Scheduler entries fire SLA_WARNING and SLA_BREACH events at exact timestamps, replacing the previous parallel-branch polling approach. Schedules auto-delete after firing. On job completion, unfired schedules are cancelled and SLA_MET is published.
• Sub-daily execution granularity: pipelines can run at hourly or daily cadence depending on sensor data. When sensors include both date and hour fields, the framework uses a composite execution date (2026-03-03T10). Glue triggers receive --par_day and --par_hour arguments automatically.
• Infrastructure trigger retry: trigger execution failures (e.g., Glue ConcurrentRunsExceededException) retry 4 times with exponential backoff (30s, 60s, 120s, 240s) via Step Functions native Retry. Each failure is logged to the joblog table for audit. This retry budget is separate from maxRetries for job failures.
• StatusChecker fallback in check-job: when no terminal joblog entry exists, the orchestrator polls the trigger API directly to determine job status.

Changed

• 18-state sequential Step Functions workflow replaces the 47-state machine. SLA monitoring uses EventBridge Scheduler instead of a parallel branch.
• sla-monitor Lambda supports 5 modes: schedule, cancel, fire-alert, calculate, reconcile.
• Declarative YAML validation rules replace the archetype/trait/evaluator system. Pipeline configs define validation as YAML rules (exists, equals, gt, gte, lt, lte, age_lt, age_gt).
• 3 DynamoDB tables (control, joblog, rerun) replace the single-table design for clearer access patterns and independent scaling.
• 4 Lambda functions (stream-router, orchestrator, sla-monitor, watchdog) replace the previous 7+ handlers.
• EventBridge events replace SNS for all alerting and lifecycle notifications.
• Reusable Terraform module — consumers deploy infrastructure without framework code in their repo.
• Trigger state retries infrastructure failures independently of job failure retries (maxRetries).

Removed

• Redis and Postgres storage providers (AWS-first; GCP and Azure planned after AWS stabilizes)
• CLI binary (cmd/interlock) and HTTP server
• Archetype, trait, and evaluator subprocess system
• Local mode (Docker Compose + Redis)

Fixed

• Pipeline config included in Step Functions execution input (#30)
• YAML configs converted to JSON before DynamoDB storage (#27)
• Sensor data data map unwrapped in stream-router (#28)
• SLA monitor handles relative :MM deadline format for hourly pipelines (#35)
• check-job skips non-terminal joblog events (#36)
• EventSLAResolved constant renamed to EventSLAMet to match published event string

v0.3.1

Fixed

• Validation timeout retry: SFN retries trait evaluation every 60s until validation timeout, instead of exiting after one attempt
• Stream-router date accuracy: reads date field from MARKER record NewImage for correct date at midnight rollover, falls back to time.Now() for backward compatibility

Added

• At-risk SLA alerting: proactive warnings before evaluation and completion deadline breaches
• Watchdog re-trigger: detects missed schedules and re-fires SFN executions for recovery
• ReleaseLock catch: ASL catches ReleaseLock errors to prevent silent lock leaks

v0.3.0

Added

• Post-Completion Monitoring Expiry: watchdog handles COMPLETED_MONITORING → COMPLETED transitions
  • CheckCompletedMonitoring() pure function scans for runs in COMPLETED_MONITORING whose monitoring window has elapsed
  • Uses RunLogEntry.UpdatedAt as the monitoring start time — no schema changes required
  • Transitions expired runs to COMPLETED via PutRunLog, appends EventMonitoringCompleted audit event
  • Monitoring window duration read from Watch.Monitoring.Duration per-pipeline config
  • Wired into both local-mode Watchdog.scan() and Lambda handler
  • Offloads the monitoring wait from Step Function executions to the watchdog's 5-minute scan, freeing SFN capacity immediately after the Glue job succeeds
  • MonitoringResult exported type for callers to inspect results
  • 3 unit tests: expired → transitions, still-in-window → skips, already-completed → ignores

v0.2.1

Fixed

• Evaluation retry loop: Step Function now retries trait evaluation every 60s when traits aren't ready, terminating via the existing validation timeout. Previously in AWS event-driven mode, the SFN wrote a PENDING RUNLOG and exited immediately — the RUNLOG stayed PENDING indefinitely unless a new MARKER arrived. The local watcher loop was unaffected.
• ResolvePipeline error logging: Step Function now writes a FAILED RUNLOG when ResolvePipeline returns an error (missing archetype, pipeline not found, archetype resolution failure). Previously these configuration errors caused a silent exit with no audit trail.
• logResult attempt counter: attemptNumber now only increments when retrying after a terminal state (COMPLETED/FAILED/CANCELLED). Transitioning within the same attempt (e.g., PENDING → COMPLETED) preserves the counter.
• Validation timeout test: Fixed midnight boundary edge case in TestCheckValidationTimeout_NotBreached where now + 1h wrapping past midnight caused a false positive.

Changed

• Step Function state count: 49 → 52 (added LogResolveFailed, CheckIfFirstEvalAttempt, WaitForReadiness)

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's New

Adds lifecycle events, stuck-run detection, and alert categorization to the interlock framework.

Lifecycle Events

The stream-router now publishes SNS events when a pipeline run reaches a terminal status (COMPLETED or FAILED). Downstream consumers — monitoring dashboards, notification systems, active recovery workflows — can subscribe to the lifecycle topic without polling DynamoDB.

• PIPELINE_COMPLETED and PIPELINE_FAILED events emitted to a dedicated lifecycle SNS topic
• Configurable via LIFECYCLE_TOPIC_ARN environment variable on stream-router Lambda
• Best-effort publishing — errors are logged, not propagated
• LifecycleEvent struct in internal/lambda/types.go

Stuck-Run Detection

The watchdog now detects runs stuck in non-terminal states (PENDING, TRIGGERING, RUNNING) beyond a configurable threshold, in addition to missed schedules.

• CheckStuckRuns() pure function scans for runs beyond threshold (default: 30 minutes)
• Dedup via distributed lock (watchdog:stuck:{pipeline}:{schedule}:{date}, 24h TTL)
• EventRunStuck event kind for audit trail
• RUN_STUCK alert with Category "stuck_run"

Alert Categorization

All alerts now carry a machine-readable Category field (JSON: alertType) for filtering and routing without parsing message strings.

• Values: schedule_missed, stuck_run, evaluation_sla_breach, completion_sla_breach, validation_timeout, trait_drift

Install

go get github.com/dwsmith1983/interlock@v0.2.0

Full Changelog: v0.1.1...v0.2.0