Skip to content

Commit 22436a5

Browse files
helloml0326helloml0326
authored
[feat]add Agent Skill graders and skills evaluation cookbook (#162)
* feat(graders): add Agent Skill evaluators and skills evaluation cookbook Introduce SkillThreatAnalysisGrader, SkillDeclarationAlignmentGrader, SkillDesignGrader, and refresh completeness/relevance graders. Remove legacy comprehensive, pairwise, safety, and structure skill graders and their tests. Add cookbooks/skills_evaluation with SkillsGradingRunner, loader models, and README. Document skill graders in docs/built_in_graders/skills.md and link from overview. Announce Skill Graders in README and README_zh. Made-with: Cursor * fix(graders): resolve pre-commit pylint and pytest failures - completeness/design/relevance: suppress W0613 for script_contents and reference_contents kept for API parity (consumed via SkillsGradingRunner) - declaration_alignment: disable too-many-lines (1138 lines) and move unused injection_fix into the findings dict (was W0612) - test_skill_completeness: catch openai.RateLimitError in consistency test and skip rather than fail - test_skill_design: gate test_accuracy_vs_expected behind RUN_ACCURACY_TESTS (strong-model-only) to prevent false failures with qwen3.5-plus; add RateLimitError skip guard to both quality tests - .pre-commit-config.yaml: use .venv/bin/python -m pytest so pre-commit picks up the project venv where pytest is installed Made-with: Cursor * style(skills): format graders, cookbook, and tests - Reorder imports and wrap SkillDeclarationAlignmentGrader imports - Reflow textwrap.dedent prompt strings in threat_analysis grader - Apply Black-style line breaks in runner, skill_models, evaluate_skills - Expand long literals in declaration_alignment and threat_analysis tests Made-with: Cursor * chore: revert pre-commit pytest hook to system python Restore `python -m pytest` instead of `.venv/bin/python -m pytest`. Made-with: Cursor
1 parent 157edd5 commit 22436a5

28 files changed

Lines changed: 6139 additions & 7186 deletions

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ It can also convert grading results into **reward signals** to help you **fine-t
5252

5353
## News
5454

55+
- **2026-04-07** - 🔒 **Skill Graders** - 5 new LLM-based graders for evaluating AI Agent Skill packages: threat analysis (AITech taxonomy), declaration alignment, completeness, relevance, and design quality. 👉 [Documentation](./docs/built_in_graders/skills.md) | [Cookbook](./cookbooks/skills_evaluation/README.md)
56+
5557
- **2026-03-10** - 🛠️ **New Skills** - Claude authenticity verification, find skills combo, and more. 👉 [Browse Skills](https://openjudge.me/skills)
5658

5759
- **2026-02-12** - 📚 **Reference Hallucination Arena** - Benchmark for evaluating LLM academic reference hallucination. 👉 [Documentation](./docs/validating_graders/ref_hallucination_arena.md) | 📊 [Leaderboard](https://openjudge.me/leaderboard)

‎README_zh.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ OpenJudge 提供**即用型评分器**，并支持生成**场景特定的评估
5050

5151
## 最新动态
5252

53+
- **2026-04-07** - 🔒 **Skill Graders** - 5 个新的基于 LLM 的 AI Agent Skill 包评估器：威胁分析（AITech 分类体系）、声明对齐、完整性、相关性和结构设计质量。 👉 [文档](./docs/built_in_graders/skills.md) | [Cookbook](./cookbooks/skills_evaluation/README.md)
54+
5355
- **2026-02-12** - 📚 **Reference Hallucination Arena** - 评估大语言模型学术引用幻觉的基准测试。 👉 [文档](./docs/validating_graders/ref_hallucination_arena.md) | 📊 [排行榜](https://openjudge.me/leaderboard)
5456

5557
- **2026-01-27** - 🖥️ **OpenJudge UI** - 基于 Streamlit 的可视化界面，支持评分器测试和 Auto Arena。👉 [在线体验](https://openjudge.me/app/) | 本地运行：`streamlit run ui/app.py`

‎cookbooks/skills_evaluation/README.md‎

Lines changed: 356 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,356 @@
1+
# Skills Evaluation Cookbook
2+
3+
使用 OpenJudge 对 AI Agent Skill 包进行多维度自动化评估。
4+
5+
---
6+
7+
## 目录
8+
9+
- [简介](#简介)
10+
- [评估维度](#评估维度)
11+
- [运行教程](#运行教程)
12+
- [报告示例](#报告示例)
13+
14+
---
15+
16+
## 简介
17+
18+
Agent Skills（技能包）是扩展 AI Agent 能力的本地目录包，每个技能包通常包含：
19+
20+
```
21+
skill-name/
22+
├── SKILL.md # 必需：YAML frontmatter + markdown 指令
23+
├── scripts/ # 可选：Agent 可执行的 Python / Bash 脚本
24+
├── references/ # 可选：指令中引用的附加文档
25+
└── assets/ # 可选：模板与资源文件
26+
```
27+
28+
`cookbooks/skills_evaluation` 提供了一套端到端的技能评估流水线，通过 **5 个独立的 LLM-as-Judge Grader** 并发对技能包打分，输出加权综合分数，并生成 JSON 与 Markdown 格式的评估报告。
29+
30+
---
31+
32+
## 评估维度
33+
34+
评估框架包含以下 5 个维度（Grader），每个维度独立运行，最终加权平均为 0–100 的综合分数。
35+
36+
### 1. Threat Analysis（威胁分析）
37+
38+
| 属性 | 说明 |
39+
|------|------|
40+
| **类** | `SkillThreatAnalysisGrader` |
41+
| **量表** | 1–4（4 = 安全，1 = 严重风险） |
42+
| **默认通过阈值** | ≥ 3（Low 或更好） |
43+
44+
基于 **AITech 分类体系**，对技能包全组件进行 LLM 语义安全扫描，涵盖：
45+
46+
- `AITech-1.1` — 直接提示注入（越狱、指令覆盖）
47+
- `AITech-1.2` — 间接提示注入（外部数据源嵌入恶意指令）
48+
- `AITech-4.3` — 协议操控 / 能力膨胀（关键词诱导、品牌冒充）
49+
- `AITech-8.2` — 数据外泄 / 暴露（硬编码凭据、未授权网络传输）
50+
- `AITech-9.1` — Agent 系统操控（命令注入、代码注入）
51+
- `AITech-9.2` — 检测规避（混淆、base64→exec 链）
52+
- `AITech-12.1` — 工具利用（工具投毒、违反 allowed-tools 限制）
53+
- `AITech-13.1` — 可用性破坏（无限循环、资源耗尽）
54+
- `AITech-15.1` — 有害 / 误导性内容
55+
56+
每个发现包含：severity 等级、AITech 分类码、证据片段、文件位置和修复建议。
57+
58+
---
59+
60+
### 2. Declaration Alignment（声明对齐）
61+
62+
| 属性 | 说明 |
63+
|------|------|
64+
| **类** | `SkillDeclarationAlignmentGrader` |
65+
| **量表** | 1–3（3 = 对齐，1 = 不匹配） |
66+
| **默认通过阈值** | ≥ 2（Uncertain 或更好） |
67+
68+
检测 `SKILL.md` 声明的功能与脚本实际行为之间的**蓄意不一致**，聚焦于：
69+
70+
- 隐藏后门、隐蔽数据管道
71+
- 未声明的网络操作（声称"本地处理"实则外传数据）
72+
- 工具投毒（description 与脚本行为不符）
73+
74+
> **注意**：若技能包没有脚本文件，该维度自动跳过并标记为通过。
75+
76+
---
77+
78+
### 3. Completeness（完整性）
79+
80+
| 属性 | 说明 |
81+
|------|------|
82+
| **类** | `SkillCompletenessGrader` |
83+
| **量表** | 1–3（3 = 完整，1 = 不完整） |
84+
| **默认通过阈值** | ≥ 2（Partially complete 或更好） |
85+
86+
评估技能包是否提供足够的细节以完成任务，检查：
87+
88+
- 步骤、输入、输出是否明确
89+
- 先决条件（环境、依赖、权限）是否说明
90+
- 错误处理与边界情况是否覆盖
91+
- 核心算法 / 公式是否正确
92+
- `SKILL.md` 承诺的能力与实现是否一致（防止"空头支票"）
93+
94+
---
95+
96+
### 4. Relevance（相关性）
97+
98+
| 属性 | 说明 |
99+
|------|------|
100+
| **类** | `SkillRelevanceGrader` |
101+
| **量表** | 1–3（3 = 完全匹配，1 = 不匹配） |
102+
| **默认通过阈值** | ≥ 2（Partial match 或更好） |
103+
104+
评估技能包与**给定任务描述**的匹配程度：
105+
106+
- 技能的核心目的是否直接完成任务（而非仅测量/评估任务结果）
107+
- 技能名称和描述是否明确定位到对应用例
108+
- 是否提供具体可操作的技术模式，而非流程脚手架
109+
110+
> 若未提供 `task_description`，将使用技能自身的 `description` 字段做自洽性检验。
111+
112+
---
113+
114+
### 5. Structure / Design（结构设计）
115+
116+
| 属性 | 说明 |
117+
|------|------|
118+
| **类** | `SkillDesignGrader` |
119+
| **量表** | 1–3（3 = 优秀，1 = 较差） |
120+
| **默认通过阈值** | ≥ 2（Partially sound 或更好） |
121+
122+
从 **7 个子维度**评估技能包的内部设计质量：
123+
124+
| 维度 | 考察点 |
125+
|------|--------|
126+
| **D1 Knowledge Delta** | 是否提供超越 Claude 基础知识的专家级内容 |
127+
| **D2 Mindset + Procedures** | 是否传授专家思维框架和非显而易见的操作流程 |
128+
| **D3 Specification Compliance** | `name` 格式是否合法；`description` 是否包含 WHAT / WHEN / KEYWORDS |
129+
| **D4 Progressive Disclosure** | 内容分层是否合理（metadata → body → references） |
130+
| **D5 Freedom Calibration** | 约束力度是否与任务脆弱性相匹配 |
131+
| **D6 Practical Usability** | 代码示例是否可用；决策树是否完整；错误处理是否有 fallback |
132+
| **D7 Anti-Pattern Quality** | 是否提供明确的 NEVER 列表（补充维度，加分项） |
133+
134+
---
135+
136+
## 运行教程
137+
138+
### 前提条件
139+
140+
安装依赖：
141+
142+
```bash
143+
pip install -r requirements.txt
144+
```
145+
146+
在项目根目录的 `.env` 文件中配置模型：
147+
148+
```env
149+
OPENAI_API_KEY=sk-...
150+
OPENAI_BASE_URL=https://your-api-endpoint/v1 # 可选，默认使用 OpenAI 官方端点
151+
OPENAI_MODEL=qwen3.6-plus # 可选，默认 qwen3.6-plus
152+
```
153+
154+
### 命令行运行
155+
156+
```bash
157+
# 评估单个技能包目录
158+
python cookbooks/skills_evaluation/evaluate_skills.py /path/to/my-skill
159+
160+
# 评估技能注册表（目录下每个子目录都是一个技能包）
161+
python cookbooks/skills_evaluation/evaluate_skills.py /path/to/skills/
162+
163+
# 附带任务描述（用于 Relevance 和 Completeness 维度）
164+
python cookbooks/skills_evaluation/evaluate_skills.py /path/to/skills/ "自动化代码审查 Pull Request"
165+
```
166+
167+
评估完成后，结果将保存到：
168+
169+
```
170+
cookbooks/skills_evaluation/results/
171+
├── grading_results.json # 结构化 JSON 报告
172+
└── grading_report.md # Markdown 可读报告
173+
```
174+
175+
### 在代码中调用
176+
177+
```python
178+
import asyncio
179+
from openjudge.models.openai_chat_model import OpenAIChatModel
180+
from cookbooks.skills_evaluation.runner import SkillsGradingRunner, build_markdown_report
181+
182+
model = OpenAIChatModel(api_key="sk-...", model="gpt-4o")
183+
184+
runner = SkillsGradingRunner(
185+
model=model,
186+
weights={
187+
"threat_analysis": 2.0, # 安全维度权重加倍
188+
"alignment": 1.5,
189+
"completeness": 1.0,
190+
"relevance": 1.0,
191+
"structure": 0.5, # 降低结构维度权重
192+
},
193+
# 自定义通过阈值（可选）
194+
thresholds={
195+
"threat_analysis": 3, # 必须达到 Low 或更好
196+
"alignment": 2,
197+
"completeness": 2,
198+
"relevance": 2,
199+
"structure": 2,
200+
},
201+
)
202+
203+
results = asyncio.run(
204+
runner.arun(
205+
"/path/to/skills/",
206+
task_description="自动化代码审查 Pull Request",
207+
)
208+
)
209+
210+
for r in results:
211+
status = "PASS" if r.passed else "FAIL"
212+
print(f"{r.skill_name}: {r.weighted_score * 100:.1f}/100 — {status}")
213+
214+
# 生成 Markdown 报告
215+
print(build_markdown_report(results))
216+
```
217+
218+
### 禁用某个维度
219+
220+
将对应维度的权重设为 `0.0` 即可跳过该维度：
221+
222+
```python
223+
runner = SkillsGradingRunner(
224+
model=model,
225+
weights={
226+
"threat_analysis": 1.0,
227+
"alignment": 0.0, # 跳过 Alignment
228+
"completeness": 1.0,
229+
"relevance": 0.0, # 跳过 Relevance
230+
"structure": 1.0,
231+
},
232+
)
233+
```
234+
235+
---
236+
237+
## 报告示例
238+
239+
以下为对 `agentic-eval` 技能包的实际评估输出。
240+
241+
### 终端输出
242+
243+
```
244+
============================================================
245+
Skill : agentic-eval
246+
Path : /workspace/OpenJudge/.agents/skills/agentic-eval
247+
Score : 0.900 ✅ PASS
248+
Time : 5.0s
249+
────────────────────────────────────────────────────────────
250+
[threat_analysis ] ✅ score=4 norm=1.00 w=1.0
251+
reason: The skill package 'agentic-eval' contains no security findings…
252+
[alignment ] ✅ score=3 norm=1.00 w=1.0
253+
reason: No scripts found; alignment check not applicable.
254+
[completeness ] ✅ score=3 norm=1.00 w=1.0
255+
reason: The skill provides clear goals, explicit steps via Python code snippets…
256+
[relevance ] ✅ score=3 norm=1.00 w=1.0
257+
reason: The skill's name, description, and content directly address the task…
258+
[structure ] ✅ score=2 norm=0.50 w=1.0
259+
reason: The skill fails significantly on Knowledge Delta (D1)…
260+
```
261+
262+
### Markdown 报告
263+
264+
---
265+
266+
# Skills Evaluation Report
267+
268+
_Total skills evaluated: **1** — Passed: **1** / 1_
269+
270+
## Summary
271+
272+
| Skill | Score | Result |
273+
|-------|------:|--------|
274+
| `agentic-eval` | 90.0 | ✅ Pass |
275+
276+
---
277+
278+
# Skill Evaluation Report: `agentic-eval`
279+
280+
> **Overall score: 90.0 / 100 — ✅ PASS** _(evaluated in 5.0s)_
281+
282+
**Path:** `.agents/skills/agentic-eval`
283+
284+
## Dimension Summary
285+
286+
| Dimension | Score | Normalised | Weight | Result |
287+
|-----------|------:|-----------:|-------:|--------|
288+
| Threat Analysis | 4 | 1.00 | 1.0 | ✅ Pass |
289+
| Alignment | 3 | 1.00 | 1.0 | ✅ Pass |
290+
| Completeness | 3 | 1.00 | 1.0 | ✅ Pass |
291+
| Relevance | 3 | 1.00 | 1.0 | ✅ Pass |
292+
| Structure | 2 | 0.50 | 1.0 | ✅ Pass |
293+
294+
## Dimension Details
295+
296+
### Threat Analysis
297+
298+
- **Score:** 4 | **Normalised:** 1.00 | **Weight:** 1.0 | **Result:** ✅ Pass
299+
300+
The skill package 'agentic-eval' contains no security findings. The YAML manifest and markdown instructions describe legitimate evaluation patterns without prompt injection, credential theft, or tool abuse.
301+
302+
### Alignment
303+
304+
- **Score:** 3 | **Normalised:** 1.00 | **Weight:** 1.0 | **Result:** ✅ Pass
305+
306+
No scripts found; alignment check not applicable.
307+
308+
### Completeness
309+
310+
- **Score:** 3 | **Normalised:** 1.00 | **Weight:** 1.0 | **Result:** ✅ Pass
311+
312+
The skill provides clear goals, explicit steps via Python code snippets for three distinct patterns (Basic Reflection, Evaluator-Optimizer, Code-Specific), and defines inputs/outputs within those examples. It addresses failure modes by including iteration limits and convergence checks.
313+
314+
### Relevance
315+
316+
- **Score:** 3 | **Normalised:** 1.00 | **Weight:** 1.0 | **Result:** ✅ Pass
317+
318+
The skill's name, description, and content directly address the task of improving AI agent outputs. It provides concrete, actionable implementation patterns with code examples that explicitly demonstrate iterative refinement loops.
319+
320+
### Structure
321+
322+
- **Score:** 2 | **Normalised:** 0.50 | **Weight:** 1.0 | **Result:** ✅ Pass
323+
324+
The skill fails on Knowledge Delta (D1) and Mindset + Procedures (D2). The content consists of generic tutorial patterns that an AI agent already knows. The description (D3) is weak, missing specific KEYWORDS and concrete trigger scenarios. Practical Usability (D6) is low because code examples rely on undefined dependencies (`llm`, `run_tests`). There is no Anti-Pattern (D7) section.
325+
326+
---
327+
328+
### JSON 报告结构
329+
330+
```json
331+
[
332+
{
333+
"skill_name": "agentic-eval",
334+
"skill_path": "/workspace/OpenJudge/.agents/skills/agentic-eval",
335+
"weighted_score": 0.9,
336+
"passed": true,
337+
"grading_duration_seconds": 5.0,
338+
"dimensions": {
339+
"threat_analysis": {
340+
"score": 4,
341+
"normalized_score": 1.0,
342+
"weight": 1.0,
343+
"reason": "The skill package contains no security findings...",
344+
"passed": true,
345+
"error": null,
346+
"metadata": { "findings": [], "threshold": 3 }
347+
},
348+
"alignment": { "score": 3, "normalized_score": 1.0, "passed": true, "..." : "..." },
349+
"completeness": { "score": 3, "normalized_score": 1.0, "passed": true, "...": "..." },
350+
"relevance": { "score": 3, "normalized_score": 1.0, "passed": true, "...": "..." },
351+
"structure": { "score": 2, "normalized_score": 0.5, "passed": true, "...": "..." }
352+
},
353+
"errors": []
354+
}
355+
]
356+
```

0 commit comments

Comments
 (0)