Skip to content

Commit f51e9e9

Browse files
authored
Add STAMP safety model mapping and pipeline patterns to README (#80)
* fix: add watchdog schedule rule for LocalStack E2E testing * docs: add STAMP safety model mapping and pipeline patterns to README The README mentioned STAMP once with no explanation. This adds a mapping table showing how each STAMP concept (controller, actuator, sensor, feedback, safety constraint) maps to Interlock components, plus four pipeline pattern examples (batch, streaming rollup, cross-pipeline dependency, ad-hoc/irregular schedule) demonstrating the sensor model as the universal interface.
1 parent 409235d commit f51e9e9

1 file changed

Lines changed: 125 additions & 2 deletions

File tree

README.md

Lines changed: 125 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Interlock
22

3-
STAMP-based safety framework for data pipeline reliability. Interlock prevents pipelines from executing when preconditions aren't safe — like a physical interlock mechanism.
3+
STAMP-based safety controller for data pipeline reliability. Interlock prevents pipelines from executing when preconditions aren't safe — like a physical interlock mechanism. Sensors report readiness, a controller evaluates safety constraints, and actuators trigger jobs only when it's safe.
44

5-
The framework applies [Leveson's Systems-Theoretic Accident Model](https://mitpress.mit.edu/9780262016629/engineering-a-safer-world/) to data engineering: pipelines have **declarative validation rules** (feedback), **sensor data in DynamoDB** (process models), and **conditional execution** (safe control actions).
5+
Built on [Leveson's Systems-Theoretic Accident Model](https://mitpress.mit.edu/9780262016629/engineering-a-safer-world/) (STAMP): pipelines have **declarative validation rules** (safety constraints), **sensor data in DynamoDB** (process models), and **conditional execution** (safe control actions).
66

77
## What Interlock Is (and Isn't)
88

@@ -34,6 +34,21 @@ Sensor data → DynamoDB Stream → stream-router Lambda → Step Functions
3434
(→ events table) (→ alert-dispatcher → Slack)
3535
```
3636

37+
### Safety Model (STAMP)
38+
39+
Interlock maps directly to STAMP's control-theoretic safety structure. Each component has a defined role in the feedback loop that prevents unsafe pipeline execution:
40+
41+
| STAMP Concept | Interlock Component | Role |
42+
|---------------|---------------------|------|
43+
| Controlled Process | User's pipeline or job | The workload being safeguarded (Glue, EMR, Airflow DAG, Databricks, etc.) |
44+
| Actuator | Trigger | Fires the job via REST call, AWS SDK, or subprocess — only when the controller says go |
45+
| Controller | orchestrator Lambda (coordinated by Step Functions) | Evaluates validation rules against sensor state; decides whether to trigger |
46+
| Sensor | DynamoDB sensor records | External processes write readiness signals (status, counts, timestamps, lag) to the control table |
47+
| Feedback | Post-run drift detection, job logs, SLA monitoring | Monitors completed jobs for late data, source drift, SLA breaches, and silent failures |
48+
| Safety Constraint | Validation rules (declarative YAML) | The preconditions that must be satisfied before the actuator fires |
49+
50+
The safety loop: sensors report the current state of upstream dependencies → the controller evaluates declarative constraints against that state → the actuator triggers the job only when all constraints pass → feedback mechanisms monitor the completed job and detect post-completion issues (drift, late data, SLA breaches) that may require a re-run.
51+
3752
### Declarative Validation Rules
3853

3954
Pipeline configs define validation as declarative YAML rules — no custom evaluator code needed:
@@ -206,6 +221,114 @@ No Step Function executions, no job triggers, no rerun requests. Remove `dryRun:
206221
| `databricks` | HTTP (REST 2.1) | Databricks job runs |
207222
| `lambda` | AWS SDK | Direct Lambda invocation |
208223

224+
## Pipeline Patterns
225+
226+
The sensor model is Interlock's universal interface. Whether your pipeline is batch, streaming rollup, ad-hoc, or depends on other pipelines — the pattern is the same: write sensor data to the control table, define validation rules, and let Interlock decide when it's safe to run.
227+
228+
The examples below show the relevant sections. A complete config also requires `pipeline:`, `job:`, and optionally `postRun:` and `dryRun:` fields — see [Pipeline Configuration](#pipeline-configuration) for a full example.
229+
230+
### Batch Precondition
231+
232+
Wait for an upstream job to report completion and a minimum row count before triggering a downstream ETL:
233+
234+
```yaml
235+
schedule:
236+
cron: "0 8 * * *"
237+
evaluation:
238+
window: 1h
239+
interval: 5m
240+
validation:
241+
trigger: "ALL"
242+
rules:
243+
- key: upstream-complete
244+
check: equals
245+
field: status
246+
value: ready
247+
- key: row-count
248+
check: gte
249+
field: count
250+
value: 1000
251+
```
252+
253+
### Streaming Rollup Safety
254+
255+
A Kafka consumer processes transactions throughout the day. At close-of-business, a batch rollup must only run when the stream has caught up. The consumer writes lag and record count sensors to the control table:
256+
257+
```yaml
258+
# schedule.trigger starts evaluation when lag drops below threshold;
259+
# validation.rules re-check at each interval until the window closes
260+
schedule:
261+
trigger:
262+
key: consumer-lag
263+
check: lte
264+
field: lag_seconds
265+
value: 30
266+
evaluation:
267+
window: 30m
268+
interval: 2m
269+
validation:
270+
trigger: "ALL"
271+
rules:
272+
- key: consumer-lag
273+
check: lte
274+
field: lag_seconds
275+
value: 30
276+
- key: record-count
277+
check: gte
278+
field: count
279+
value: 5000
280+
- key: cutoff-status
281+
check: equals
282+
field: status
283+
value: closed
284+
```
285+
286+
### Cross-Pipeline Dependency
287+
288+
Upstream pipeline handlers write success sensors directly to the downstream pipeline's control table entry (`PK = PIPELINE#<downstream-id>`). No special cross-pipeline machinery — it's just a sensor write to the right partition key:
289+
290+
```yaml
291+
# silver-daily pipeline — waits for all 24 hourly runs to complete
292+
schedule:
293+
trigger:
294+
key: daily-status
295+
check: equals
296+
field: all_hours_complete
297+
value: true
298+
validation:
299+
trigger: "ALL"
300+
rules:
301+
- key: daily-status
302+
check: equals
303+
field: all_hours_complete
304+
value: true
305+
```
306+
307+
See [interlock-aws-example](https://github.com/dwsmith1983/interlock-aws-example) for the full bronze → silver-hourly → silver-daily dependency chain.
308+
309+
### Ad-Hoc / Irregular Schedule
310+
311+
For pipelines that run on specific business dates (month-end close, quarterly reporting) rather than a fixed cron. Use an inclusion calendar with a relative SLA measured from first sensor arrival:
312+
313+
```yaml
314+
schedule:
315+
include:
316+
dates:
317+
- "2026-01-31"
318+
- "2026-02-28"
319+
- "2026-03-31"
320+
trigger:
321+
key: month-end-ready
322+
check: equals
323+
field: status
324+
value: ready
325+
evaluation:
326+
window: 8h
327+
interval: 10m
328+
sla:
329+
maxDuration: 4h
330+
```
331+
209332
## Deployment
210333

211334
Interlock ships as a **reusable Terraform module** — no framework code in your repo.

0 commit comments

Comments
 (0)