Repo: https://github.com/snehasish01/bathtub-fmea · Paper (Zenodo): 10.5281/zenodo.21607993
Project Bathtub applies classical Failure Mode and Effects Analysis (FMEA) — a reliability-engineering discipline with roots in 1940s aerospace and later formalized for automotive/aerospace (AIAG-VDA) — to a domain that doesn't yet have this kind of structured hazard prioritization: autonomous LLM coding agents.
This repo holds Phase 0: an 80-item failure-mode taxonomy across 10 categories, each scored
on Severity, Occurrence, and Detectability (1–10) and ranked by Risk Priority Number
(RPN = S × O × D).
📄 Read the paper (also on Zenodo) · 📊 Explore the data
Ranking all 80 failure modes by RPN puts verification/self-assessment and specification/communication failures at or above every hallucination-related failure mode — the opposite of where most current agent-tooling discourse focuses. Hallucinated APIs are real, but they're also the cheapest to detect: a compiler or import resolver catches most of them immediately. An agent claiming it ran tests it never ran, or silently resolving an ambiguous spec the wrong way, is both harder to detect and just as costly.
| Category | # Modes | Avg RPN | Max RPN | Phase-1 priority |
|---|---|---|---|---|
| A. Specification & Intent | 7 | 216 | 336 | Medium |
| B. Knowledge & Hallucination | 9 | 123 | 196 | High (cheap to instrument) |
| C. Code Correctness | 13 | 200 | 336 | High |
| D. Environment & Tooling | 7 | 76 | 120 | Medium |
| E. Agentic Execution (multi-step) | 12 | 162 | 294 | Critical |
| F. Context Management | 6 | 148 | 216 | Medium |
| G. Tool Use | 6 | 108 | 245 | High |
| H. Verification & Self-Assessment | 7 | 211 | 336 | Critical |
| I. Security & Compliance | 8 | 167 | 240 | High (off-the-shelf tools) |
| J. Communication & Reporting | 5 | 219 | 294 | Low initially |
| Total | 80 | 164 | 336 |
Four rows are flagged as research-critical — direct inputs to Phase 2 hazard modeling, not just mitigation targets:
- E1 (error compounding) and E9 (verification-step decay) — the two hypothesized drivers of a rising "wear-out" hazard rate over a long agent run, the bathtub-curve analogy this project is named for.
- H1 (false verification claims) and H4 (overconfident assertion) — the flagship product-facing checks: is a verification claim backed by a logged execution, and is stated confidence calibrated to actual correctness.
bathtub-fmea/
├── paper/
│ ├── bathtub_paper.pdf # Phase 0 report (preprint-style writeup)
│ └── bathtub_paper.tex # LaTeX source
├── data/
│ ├── bathtub_phase0_fmea.xlsx # Full workbook: README, Rating Scales, FMEA, Priority Summary
│ └── bathtub_phase0_fmea.csv # Flat CSV of the 80-item FMEA table
├── figures/
│ ├── chart_rpn_by_category.png
│ └── chart_top15_rpn.png
└── README.md
The workbook's FMEA sheet computes RPN with a live formula (=S*O*D) and includes a
conditional color scale, so editing any Severity/Occurrence/Detectability score automatically
re-ranks the taxonomy. The Rating Scales sheet documents the 1–10 anchors for all three axes.
Severity and Detectability can mostly be reasoned about from first principles. Occurrence cannot — it's a claim about how often each failure mode actually happens across real agent sessions, and this repo does not yet contain that measurement. Every Occurrence score here is a hypothesis, informed by hands-on agent experience and the empirical literature cited in the paper, flagged explicitly for revision once real telemetry exists. Closing this gap is the entire point of Phase 1.
| Phase | Goal |
|---|---|
| 0 (this repo) | Structured failure-mode taxonomy + RPN prioritization |
| 1 | Instrumentation — log the "Phase-1 Signal" field embedded per failure mode |
| 2 | Hazard modeling — fit h(step) (e.g. Cox proportional hazards) against step-indexed logs, test the E1/E9 hypothesis, recalibrate Occurrence from real data |
| 3 | Interface — surface calibrated reliability data back to users (empirical confidence in place of model self-report; automated claim-vs-execution auditing) |
See CITATION.cff, or cite directly:
Satpathy, S. (2026). Project Bathtub: A Reliability-Engineering Approach to
Failure-Mode Taxonomy for AI Coding Agents (Phase 0 — FMEA). Zenodo.
https://doi.org/10.5281/zenodo.21607993
- Code and data (
data/, chart-generation scripts): MIT - Paper text and figures: CC BY 4.0 — reuse with attribution
This is a Phase 0 artifact and the Occurrence column especially benefits from outside challenge. Issues and PRs proposing score revisions, additional failure modes, or citations to supporting/contradicting empirical work are welcome.