DARWIN-PHOENIX

DARWIN-PHOENIX is a co-evolutionary LLM agent framework for adversarially robust code generation. A Generator agent (DARWIN) iteratively rewrites code to resist failures generated by an adversarial Breaker agent (PHOENIX), creating a controlled arms race environment. The system measures behavioral fingerprinting — TF-IDF cosine distance between Round 1 and Round N code — as a novel signal for generator strategy drift under adversarial pressure.

Central finding: Behavioral drift under adversarial co-evolution strongly predicts code quality degradation. Degraded tasks exhibit 2.6× greater maximum fingerprint drift than successful ones (Kruskal-Wallis p=0.0003), and drift increases monotonically with round number (Spearman ρ=0.720, p<0.0001).

Author: Saketh Yalamanchili — M.S. Data Science & Analytics, Florida Atlantic University

---

Research Question

Is adversarial pressure or failure exposure the primary mechanism behind robustness improvement in co-evolving LLM code generation pipelines? And can behavioral drift be used to predict quality degradation before it happens?

---

System Architecture

A deterministic LangGraph StateGraph with 7 nodes executing sequentially each round:

initialize → generator → breaker → executor → scorer → evolver → terminator
                ↑                                                      |
                └──────────────────── "loop" ──────────────────────────┘
                                                                 "done" → END

Node	Agent	Role
initialize_node	—	Load HumanEval+ task, canonical tests, condition config
generator_node	DARWIN	Write/refine Python using Qwen3-32B; adapt via strategy prefix each round
breaker_node	PHOENIX	Generate adversarial tests (overflow, unicode, type confusion, boundary, deep nesting)
executor_node	—	Subprocess sandbox; compute pass@k, adversarial pass@k, bug rate
scorer_node	—	Compute AF score: 0.35·pass_can + 0.35·pass_adv + 0.20·adv_ratio + 0.10·ΔAF
evolver_node	Llama-3.1-8B-Instant	Update generator defense heuristics and breaker attack vectors
terminator_node	—	7-gate deterministic classifier → correct / degraded / antifragile

---

Experimental Design

Four conditions isolate the mechanism variable:

Condition	Name	Behavior
A	Baseline	Single-pass generation; no adversarial testing
B	Failure Corpus	Generator augmented with static failure examples; no live adversary
C	Full Co-Evolution	DARWIN + PHOENIX both evolve each round (true arms race)
D	Frozen Adversary	Breaker active Round 1 only; tests frozen thereafter

---

Final Results

Experiment 1 — Behavioral Fingerprint Drift (n=50 tasks, Condition C)

Round	N	Mean Distance	SD	Median
1 (baseline)	49	0.000	0.000	0.000
2	42	0.162	0.115	0.148
3	31	0.174	0.115	0.132
4	22	0.201	0.151	0.156

• Spearman ρ=0.720, p<0.0001 — drift increases monotonically with round
• Degraded tasks: mean max drift 0.240 vs. 0.092 for correct tasks (2.6×, p=0.0003)

Experiment 2 — Code Quality Across Conditions (n=164 tasks × 4 conditions = 656 runs)

Condition	Correct	Degraded	Degraded%	Mean AF
A — Baseline	148	16	9.8%	0.048
B — Corpus	146	18	11.0%	0.025
C — Co-Evol	153	11	6.7%	0.024
D — Frozen	152	12	7.3%	0.019

• Ordering C < D < A < B — consistent, zero reversals across all condition pairs
• Fisher exact C vs B: OR=0.58, p=0.121 (directional pilot; underpowered at 49%)
• Kruskal-Wallis on AF scores: H=7.537, p=0.057

Experiment 3 — Fault Recovery Under Injected Failures (n=50 tasks × 2 conds × 3 faults = 300 runs)

Fault Type	Cond A	Cond C	Δ	Cohen h
Hallucination	76.0%	80.0%	+4.0 pp	0.097
Context overflow	78.0%	86.0%	+8.0 pp	0.209
Timeout	84.0%	82.0%	−2.0 pp	−0.053
Overall	79.3%	82.7%	+3.3 pp	0.085

• Hallucination step distributions: Mann-Whitney U=903, p=0.010
• Fault type moderates recovery difficulty: K-W H=10.498, p=0.005

---

Paper

Research paper available in this repository:

File	Description
PAPER.tex	Full LaTeX source (arXiv-ready)
PAPER.md	Markdown version
paper/darwin_phoenix_arxiv.pdf	Paper PDF
paper/darwin_phoenix_arxiv.zip	arXiv submission package (PAPER.tex + all figures)

Title: DARWIN-PHOENIX: Behavioral Drift Under Adversarial Co-Evolution Predicts Code Quality Degradation in LLM Code Generation

Blog: I Built Two AI Agents That Fight Each Other to Write Better Code — Here's What I Found

---

Repository Structure

DARWIN-PHOENIX/
├── state.py                        # DPState TypedDict — full graph state schema
├── graph.py                        # LangGraph StateGraph + route_verdict()
├── prompts.py                      # Generator (Round 0 / Round N) + Breaker prompts
├── nodes/
│   ├── llm_client.py               # Shared LLM factory (Groq / OpenRouter)
│   ├── initialize.py               # N1: task loading, corpus injection
│   ├── generator.py                # N2: DARWIN — Qwen3-32B code generation
│   ├── breaker.py                  # N3: PHOENIX — adversarial test generation
│   ├── executor.py                 # N4: subprocess sandbox, pass@k computation
│   ├── scorer.py                   # N5: AF score formula, Bandit, coverage
│   ├── evolver.py                  # N6: Llama-3.1-8B strategy evolution
│   └── terminator.py               # N7: 7-gate deterministic classifier
├── experiments/
│   ├── exp1_runner.py              # 164 tasks × 4 conditions (656 runs)
│   ├── exp2_chaos.py               # Fault injection stress test (300 runs)
│   ├── exp3_fingerprint.py         # Behavioral fingerprinting (50 tasks, Cond C)
│   └── exp3_analysis.py            # Statistical analysis → exp3_statistical_report.txt
├── figures/
│   ├── generate_figures.py         # Generates fig1–fig4 from results CSVs
│   ├── generate_architecture.py    # Generates system architecture figure
│   ├── fig_architecture.png        # Figure 1 — System architecture
│   ├── fig1_fingerprint.png        # Figure 2 — Behavioral fingerprint drift
│   ├── fig2_quality.png            # Figure 3 — Code quality across conditions
│   ├── fig3_recovery.png           # Figure 4 — Fault recovery
│   └── fig4_summary.png            # Appendix — Combined summary
├── results/
│   ├── exp1_results.csv            # Exp 1 final results (656 runs)
│   ├── exp2_results.csv            # Exp 2 final results (300 runs)
│   ├── exp3_results.csv            # Exp 3 final results (50 tasks × 4 rounds)
│   ├── exp3_fingerprint.jsonl      # Raw fingerprint data
│   ├── exp1_statistical_report.txt # Full statistical analysis
│   ├── exp2_statistical_report.txt
│   └── exp3_statistical_report.txt
├── data/
│   ├── failure_corpus.json         # Curated failure examples (Condition B)
│   └── probe_tasks.json            # Fixed probe tasks (Exp 3)
├── PAPER.tex                       # Full paper (LaTeX, arXiv-ready)
├── PAPER.md                        # Full paper (Markdown)
├── darwin_phoenix_arxiv.zip        # arXiv submission zip
├── analysis.py                     # Supplementary analysis + visualization
├── verify_results.py               # Post-run sanity checks on all CSVs
├── smoke_test.py                   # Pre-scale validation (5 tasks × 4 conds)
├── verbose_run.py                  # Single-task full lifecycle trace
├── watchdog_exp2.py                # Experiment 2 watchdog runner
├── watchdog_exp3.py                # Experiment 3 watchdog runner
├── watchdog_exp3.ps1               # PowerShell watchdog (Windows)
├── Dockerfile.sandbox              # dp-sandbox Docker image
├── requirements.txt
├── setup.sh
└── docs/
    └── MASTERBOOK.md               # Full academic design + implementation reference

---

Setup

pip install uv
uv venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # Linux/Mac
uv pip install -r requirements.txt

Copy .env.example to .env and set your API key:

LLM_PROVIDER=openrouter
OPENROUTER_API_KEY=your_key_here
GENERATOR_MODEL=qwen/qwen3-32b
BREAKER_MODEL=qwen/qwen3-32b
EVOLVER_MODEL=meta-llama/llama-3.1-8b-instant

Docker Sandbox (optional — required for G6 branch coverage gate)

docker build -t dp-sandbox -f Dockerfile.sandbox .

Validate

python verbose_run.py HumanEval/0   # full trace on one task
python smoke_test.py                # 5 tasks × 4 conditions

---

Reproduce Results

# Experiment 1 — code quality across conditions
python experiments/exp1_runner.py --max-rounds 5

# Experiment 2 — fault injection
python experiments/exp2_chaos.py

# Experiment 3 — behavioral fingerprinting
python experiments/exp3_fingerprint.py

# Regenerate figures
python figures/generate_figures.py
python figures/generate_architecture.py

# Verify results integrity
python verify_results.py

All runners support --retry-errors to re-run failed rows without touching successful ones.

---

LLM Stack

Role	Model	Provider	Wall-clock timeout
Generator (DARWIN)	qwen/qwen3-32b	OpenRouter	240s
Breaker (PHOENIX)	qwen/qwen3-32b	OpenRouter	240s
Evolver	meta-llama/llama-3.1-8b-instant	OpenRouter	60s

enable_thinking: False on Qwen3-32B disables chain-of-thought (10× latency reduction). Exponential backoff: base 2s, max 120s, 6 outer retries.

---

Full academic mechanics, node implementation details, prompt engineering, and design rationale: docs/MASTERBOOK.md