feat(ai-phase3): add model-routing + llm-cost-optimization; update ai-harness + ai-product-design; agent skill refs (v1.9.0)

Author: kienbui1995

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "magic-powers",
   "description": "Cost-optimized Claude Code plugin — 11 built-in agents (+ 37 optional cloud/amplitude agents), 178 workflow skills, and 12 slash commands. Built for AI startups shipping fast with small teams.",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "author": {
     "name": "kienbui1995"
   },

CHANGELOG.md

@@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.9.0] — 2026-04-10
+
+### Added
+- 🔀 **`model-routing`** — task-based routing, cascade escalation (cheap→expensive), content-based routing, fallback chains, routing metrics
+- 💰 **`llm-cost-optimization`** — prompt caching (42% reduction), output token reduction, batch API (50% discount), workflow cost accounting, application-level caching
+
+### Updated
+- **`ai-harness`**: statistical rigor (t-test, Cohen's d, sample sizes), dataset maintenance (stale detection, production failure ingestion)
+- **`ai-product-design`**: agent UX patterns (progress display, transparent tool use), citation/attribution for RAG, accessibility (ARIA, mobile, i18n)
+
+### Agents updated
+- `@ai-engineer`: added `llm-cost-optimization` + `model-routing`
+- `@ai-product`: added `model-routing`
+
+---
+
 ## [1.8.0] — 2026-04-10
 
 ### Added

README.md

@@ -8,7 +8,7 @@
 [![Commands](https://img.shields.io/badge/commands-12-blue)](README.md#12-slash-commands)
 [![Website](https://img.shields.io/badge/website-magic--powers.pmai.space-orange)](https://magic-powers.pmai.space)
 
-**[magic-powers.pmai.space](https://magic-powers.pmai.space)** — A Claude Code plugin with cost-optimized model routing, 43 built-in skills + 67 optional skills + 42 cloud division skills (193 total), 11 specialized agents, 12 slash commands, and optional hooks/MCP integration. Built for all professional roles — dev, product, data, marketing, sales, design, and more. Also works with Cursor, Copilot, Aider, Windsurf, Gemini CLI, Codex, Kiro, and OpenCode.
+**[magic-powers.pmai.space](https://magic-powers.pmai.space)** — A Claude Code plugin with cost-optimized model routing, 43 built-in skills + 67 optional skills + 42 cloud division skills (195 total), 11 specialized agents, 12 slash commands, and optional hooks/MCP integration. Built for all professional roles — dev, product, data, marketing, sales, design, and more. Also works with Cursor, Copilot, Aider, Windsurf, Gemini CLI, Codex, Kiro, and OpenCode.
 
 ## Why Magic Powers?
 
@@ -191,7 +191,7 @@ Skills ported from [amplitude/mcp-marketplace](https://github.com/amplitude/mcp-
 | Sonnet | 6 (debugger, db-optimizer, sre, git-workflow, ui-designer, product-strategist) | $$ |
 | Haiku | 4 (reviewer, security-reviewer, technical-writer, copywriter) | $ |
 
-## 193 Skills
+## 195 Skills
 
 **Core Workflow:** `using-magic-powers` · `brainstorming` · `writing-plans` · `executing-plans`
 

agents/ai/engineer.md

@@ -16,6 +16,8 @@ skills:
   - magic-powers:ai-safety-guardrails
   - magic-powers:agentic-security
   - magic-powers:ai-harness
+  - magic-powers:llm-cost-optimization
+  - magic-powers:model-routing
 ---
 
 You are an AI engineer specializing in building production-grade AI features — from LLM integration to agentic systems to evaluation infrastructure.

agents/ai/product.md

@@ -12,6 +12,7 @@ skills:
   - magic-powers:llm-evaluation
   - magic-powers:prompt-engineering
   - magic-powers:llm-observability
+  - magic-powers:model-routing
 ---
 
 You are an AI product specialist helping teams ship AI features that users trust and love.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "magic-powers",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "description": "Cost-optimized Claude Code plugin — 43 skills, 11 agents, 9-tool support. Built for AI startups shipping fast with small teams.",
   "type": "module",
   "bin": {

site/index.html

@@ -4,7 +4,7 @@
   <meta charset="UTF-8">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">
   <title>✨ Magic Powers — Cost-optimized AI coding plugin</title>
-  <meta name="description" content="11 specialized agents, 193 workflow skills (incl. GCP/AWS/Azure cloud divisions), 9-tool support. ~75% cost reduction with no quality loss. Built for AI startups shipping fast with small teams.">
+  <meta name="description" content="11 specialized agents, 195 workflow skills (incl. GCP/AWS/Azure cloud divisions), 9-tool support. ~75% cost reduction with no quality loss. Built for AI startups shipping fast with small teams.">
   <style>
     * { margin: 0; padding: 0; box-sizing: border-box; }
 
@@ -225,10 +225,10 @@
 </nav>
 
 <div class="hero">
-  <div class="badge">v1.8.0 · <span>MIT License</span> · npm install magic-powers</div>
+  <div class="badge">v1.9.0 · <span>MIT License</span> · npm install magic-powers</div>
   <h1>Stop paying for<br>AI that guesses</h1>
   <p class="subtitle">
-    11 specialized agents routed to the right model. 193 workflow skills.
+    11 specialized agents routed to the right model. 195 workflow skills.
     9 tools supported. Built for AI startups shipping fast with small teams.
   </p>
   <div class="hero-actions">
@@ -261,7 +261,7 @@ <h1>Stop paying for<br>AI that guesses</h1>
     <div class="stat-label">Specialized agents</div>
   </div>
   <div class="stat">
-    <div class="stat-num">193</div>
+    <div class="stat-num">195</div>
     <div class="stat-label">Workflow skills</div>
   </div>
   <div class="stat">

skills/ai-harness/SKILL.md

@@ -120,6 +120,73 @@ winner = select_winner(results, primary_metric="accuracy", cost_constraint=1.2)
 - Running eval only before major releases (too late to catch drift)
 - One aggregated score hiding category regressions
 
+## Statistical Rigor
+
+Avoid declaring "improvement" without statistical evidence:
+
+```python
+from scipy import stats
+
+def is_statistically_significant(baseline_scores, new_scores, alpha=0.05):
+    """Two-sample t-test for eval score comparison"""
+    t_stat, p_value = stats.ttest_ind(baseline_scores, new_scores)
+    
+    effect_size = (mean(new_scores) - mean(baseline_scores)) / std(baseline_scores)
+    
+    return StatResult(
+        significant=p_value < alpha,
+        p_value=p_value,
+        effect_size=effect_size,  # Cohen's d
+        practical_significant=abs(effect_size) > 0.2,  # small effect
+        recommendation="ship" if (p_value < alpha and effect_size > 0.2) else "no change"
+    )
+
+# Minimum sample sizes for reliable conclusions:
+# Small effect (d=0.2): n ≥ 197 per group
+# Medium effect (d=0.5): n ≥ 52 per group
+# Large effect (d=0.8): n ≥ 26 per group
+```
+
+**Key rules:**
+- Never compare single-run scores — always run 5-10 times minimum
+- Report confidence intervals, not just averages
+- Distinguish statistical significance (p<0.05) from practical significance (effect size matters)
+- A 1% quality improvement usually isn't worth shipping complexity
+
+## Dataset Maintenance
+
+Golden datasets degrade — keep them fresh:
+
+```python
+class EvalDatasetManager:
+    def review_test_cases(self, dataset: Dataset, threshold_days=30):
+        stale = [tc for tc in dataset if tc.last_reviewed_days > threshold_days]
+        low_signal = [tc for tc in dataset if tc.pass_rate in (0.0, 1.0)]
+        # Pass rate 0% = always fails (broken test or impossible), 1% = always passes (too easy)
+        
+        return DatasetHealthReport(
+            stale_count=len(stale),
+            low_signal_count=len(low_signal),
+            action="review_and_update" if len(stale) + len(low_signal) > len(dataset) * 0.2 else "ok"
+        )
+    
+    def add_from_production_failures(self, prod_failures: list[Failure]):
+        """Convert production failures into eval test cases"""
+        for failure in prod_failures:
+            if failure.confirmed_bug:  # human verified
+                dataset.add(TestCase(
+                    input=failure.input,
+                    expected=failure.expected_output,
+                    source="production_failure",
+                    added_date=today()
+                ))
+```
+
+**Dataset health signals:**
+- >20% of tests always pass → too easy, add harder cases
+- >10% of tests never pass → broken tests or capability gap, investigate
+- Last reviewed >30 days → likely stale, review against recent prod distribution
+
 ## Integration
 
 - Use with `llm-evaluation` (frameworks) and `llm-observability` (production monitoring)

skills/ai-product-design/SKILL.md

@@ -104,13 +104,124 @@ Quality:
   Error rate by type (timeout, refusal, safety)
 ```
 
+## Agent UX Patterns
+
+Showing agent work to users requires different UX than single-call AI:
+
+**Displaying agent progress:**
+```
+┌─────────────────────────────────────────┐
+│ 🤖 Researching your question...         │
+│                                         │
+│ ✅ Searched documentation (0.8s)        │
+│ ✅ Found 3 relevant sections (0.3s)     │
+│ ⏳ Analyzing and synthesizing...        │
+│                                         │
+│ Step 3 of 4 — ~10s remaining           │
+└─────────────────────────────────────────┘
+```
+
+**Agent UX principles:**
+- Show steps in real-time (users trust agents they can see working)
+- Reveal tools used: "I searched 3 sources" builds transparency
+- Show intermediate results progressively (don't wait until full completion)
+- Provide cancel option — users need control over long-running agents
+- On failure: explain what was tried before failing ("I searched docs, tried 2 approaches, couldn't find...")
+
+**Displaying tool use transparently:**
+```
+Agent used: 🔍 web_search("Q3 revenue report 2024")
+Agent used: 📄 read_file("annual_report.pdf")
+Agent used: 🧮 calculate(formula="revenue * 0.15")
+```
+Collapsible by default — show on hover/expand for curious users.
+
+## Citation & Attribution
+
+For RAG-generated content, attribution builds trust:
+
+```
+Answer: The product launch is scheduled for Q2 2025.
+
+Sources used:
+  [1] Product Roadmap 2025.pdf — p.3: "Q2 2025 launch target"
+  [2] Engineering Timeline.xlsx — Sheet: Milestones
+```
+
+**Implementation:**
+```javascript
+// Track citations during RAG generation
+const response = await generateWithCitations(query, retrievedChunks);
+// response: { answer: "...", citations: [{source, page, quote}] }
+
+// Render with inline citation markers
+renderWithCitations(response.answer, response.citations);
+// → "The launch is scheduled for Q2 [1]" with expandable citation [1]
+```
+
+**When to show citations:**
+- Always for factual claims (dates, numbers, names)
+- Always for medical/legal/financial content
+- Optional for stylistic/creative responses
+- Always when user can verify the underlying source
+
 ## Key Outputs
 
 - UX spec: streaming behavior, loading states, error messages, fallback flow
 - Reliability design: circuit breaker thresholds, timeout values, caching strategy
 - Disclosure copy: AI labels, opt-out mechanism, data usage statement
 - Metrics dashboard definition: acceptance rate, task completion, regeneration rate
 
+## Accessibility & Mobile
+
+**Accessibility for AI-specific UI elements:**
+```javascript
+// Streaming text — announce completion to screen readers
+<div 
+  role="log"               // live region for screen readers
+  aria-live="polite"       // don't interrupt, announce when idle
+  aria-label="AI response"
+>
+  {streamingText}
+</div>
+
+// Loading state — meaningful label, not just spinner
+<div role="status" aria-label="AI is generating a response, please wait">
+  <Spinner />
+  <span className="sr-only">Generating response...</span>
+</div>
+
+// Confidence indicator — explain what percentage means
+<span 
+  title="AI confidence: 85% - This answer is likely accurate but verify for important decisions"
+  aria-label="85% confidence"
+>
+  ●●●●○
+</span>
+```
+
+**Mobile patterns:**
+- Streaming on mobile: throttle updates to 100ms (not every token) — reduces reflow
+- Loading states: skeleton screens better than spinners on small screens
+- Error messages: use native alert dialogs (not toasts) on mobile — more visible
+- Latency expectation: mobile users tolerate 5-8s more than desktop users (2-3s)
+- Offline handling: queue AI requests when offline, process when reconnected
+
+**Internationalization of AI responses:**
+```javascript
+// AI error messages need translation
+const AI_ERROR_MESSAGES = {
+  rate_limit: t('ai.error.rateLimitMessage'),        // localized
+  timeout: t('ai.error.timeoutMessage'),
+  safety_refusal: t('ai.error.safetyMessage'),
+};
+
+// RTL support for streaming text
+<div dir="auto" lang={userLocale}>  {/* auto-detects RTL */}
+  {streamingText}
+</div>
+```
+
 ## Anti-Patterns
 
 - Blocking UI while waiting for AI response (always show progress)