feat: add score command by adamaltman · Pull Request #2648 · Redocly/redocly-cli

adamaltman · 2026-03-12T01:21:13Z

What/Why/How?

What: Adds a new score command to Redocly CLI that analyzes OpenAPI 3.x descriptions and produces two composite scores: Integration Simplicity (0-100, how easy is this API to integrate) and Agent Readiness (0-100, how usable is this API by AI agents/LLM tooling).

Why: API producers currently lack a quick, deterministic way to assess how developer-friendly or AI-agent-friendly their API descriptions are. The existing stats command counts structural elements but doesn't evaluate quality signals like documentation coverage, example presence, schema complexity, or error response structure. This command fills that gap with actionable, explainable scores.

How: The implementation follows the same pattern as the stats command (bundle + analyze), with a clean separation between metric collection and score calculation:

Metric collection (collectors/): Walks the bundled document, resolving internal $refs, to gather per-operation raw metrics (parameter counts, schema depth, polymorphism, description/constraint/example coverage, structured error responses, workflow dependency depth via shared schema refs).
Scoring (scoring.ts): Pure functions that normalize raw metrics into subscores and compute weighted composite scores. Thresholds and weights are configurable constants. anyOf is penalized more heavily than oneOf/allOf; discriminator presence improves the agent readiness polymorphism clarity subscore.
Hotspots (hotspots.ts): Identifies the operations with the most issues, sorted by number of reasons, with human-readable explanations.
Output: --format=stylish (default, with color bar charts) and --format=json (machine-readable for CI/dashboards).

Reference

Related to API governance and developer experience tooling. No existing issue -- this is a new feature.

Testing

43 unit tests across 3 test files covering schema depth calculation, $ref resolution, polymorphism counting (oneOf/anyOf/allOf), constraint detection (including const), example coverage scoring, anyOf penalty multiplier, discriminator impact on agent readiness, deterministic output, and score range validation.
Manually tested against three real OpenAPI descriptions (Redocly Cafe: 12 operations, Reunite Main: 299 operations, Rebilly: 606 operations) to verify scores are reasonable and hotspot reasoning is actionable.
TypeScript compiles cleanly (tsc --noEmit), all existing tests continue to pass.

Screenshots (optional)

Stylish output for Redocly Cafe:

  Scores

  Integration Simplicity:  85.3/100
  Agent Readiness:         94.4/100

  Integration Simplicity Subscores

  Parameter Simplicity     [█████████████████░░░] 83%
  Schema Simplicity        [████████████░░░░░░░░] 62%
  Documentation Quality    [███████████████████░] 97%
  Constraint Clarity       [███████████████████░] 96%
  Example Coverage         [██████████████████░░] 92%
  Error Clarity            [████████████████████] 100%
  Workflow Clarity         [█████████████████░░░] 83%

  Top 4 Hotspot Operations

  GET /orders (listOrders)
    Integration Simplicity: 69.1  Agent Readiness: 93.9
    - High parameter count (6)
    - Deep schema nesting (depth 5)

  PATCH /orders/{orderId} (updateOrder)
    Integration Simplicity: 77.6  Agent Readiness: 85.3
    - Missing request body examples

Check yourself

Code changed? - Tested with Redoc/Realm/Reunite (internal)
All new/updated code is covered by tests
New package installed? - Tested in different environments (browser/node)
Documentation update considered

Security

The security impact of the change has been considered
Code follows company security practices and guidelines

changeset-bot · 2026-03-12T01:21:17Z

🦋 Changeset detected

Latest commit: 92d40f7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@redocly/cli	Minor
@redocly/openapi-core	Minor
@redocly/respect-core	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-03-12T01:23:24Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	80.18% (🎯 79%)	6841 / 8531
🔵	Statements	79.63% (🎯 78%)	7087 / 8899
🔵	Functions	83.1% (🎯 82%)	1372 / 1651
🔵	Branches	71.79% (🎯 71%)	4604 / 6413

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/cli/src/commands/score/constants.ts	100%	100%	100%	100%
packages/cli/src/commands/score/hotspots.ts	100%	97.67%	100%	100%
packages/cli/src/commands/score/index.ts	100%	83.33%	100%	100%
packages/cli/src/commands/score/scoring.ts	100%	93.33%	100%	100%
packages/cli/src/commands/score/collectors/document-metrics.ts	90.83%	80.35%	100%	95.41%	177, 191, 210, 229, 242, 266, 297, 301, 304-305, 308, 343
packages/cli/src/commands/score/collectors/workflow-graph.ts	100%	90%	100%	100%
packages/cli/src/commands/score/formatters/json.ts	100%	100%	100%	100%
packages/cli/src/commands/score/formatters/stylish.ts	100%	83.33%	100%	100%

Generated in workflow #8981 for commit 6cd3af4 by the Vitest Coverage Report Action

github-actions · 2026-03-12T01:23:44Z

CLI Version	Mean Time ± Std Dev (s)	Relative Performance (Lower is Faster)
cli-latest	3.469s ± 0.036s	▓ 1.00x
cli-next	3.462s ± 0.021s	▓ 1.00x (Fastest)

.changeset/add-score-command.md

docs/@v2/commands/index.md

docs/@v2/commands/score.md

packages/cli/src/commands/score/collectors/document-metrics.ts

- Add "AI" before "agent readiness" in changeset and docs for clarity - Replace <pre> block with fenced code block in score.md - Add security scheme coverage to metrics documentation - Remove resolveIfRef helper, replace with resolveNode that falls back to the original node when resolution fails - Refactor to use walkDocument visitor approach (matching stats command pattern) instead of manually iterating the document tree - Use resolveDocument + normalizeVisitors + walkDocument from openapi-core for proper $ref resolution and spec-format extensibility - Update index.test.ts to mock the new walk infrastructure Made-with: Cursor

Made-with: Cursor

docs/@v2/commands/score.md

.changeset/add-score-command.md

Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>

tatomyr

Left a couple of comments. I haven'd fully reviewed the scoring and collectors though as it takes time.

docs/@v2/commands/score.md

packages/cli/src/commands/score/__tests__/document-metrics.test.ts

packages/cli/src/commands/score/__tests__/formatters.test.ts

docs/@v2/commands/score.md

packages/cli/src/commands/score/index.ts

tatomyr

I found some dead code. Please check if it's needed and remove if not. Also, I'm not sure what tests to review as many appear to only test that dead code, so let's handle that first.

tatomyr · 2026-03-16T10:00:33Z

packages/cli/src/commands/score/__tests__/collect-metrics.ts

@@ -0,0 +1,82 @@
+import {


Is it code or tests? It code, then it should be outside the __tests__ folder.

This code doesn't seems to be used anywhere except in tests. Please consider removing it and the tests.

tatomyr · 2026-03-16T10:12:41Z

packages/cli/src/commands/score/collectors/document-metrics.ts

+}
+
+export function resetSchemaWalkState(s: SchemaWalkState): void {
+  s.depth = -1;


What does the negative depth mean?

tatomyr · 2026-03-16T10:17:40Z

packages/cli/src/commands/score/__tests__/document-metrics.test.ts

+import { parseYaml } from '@redocly/openapi-core';
+import { outdent } from 'outdent';
+
+import { collectDocumentMetrics } from './collect-metrics.js';
+
+function yaml(source: string): Record<string, unknown> {
+  return parseYaml(source) as Record<string, unknown>;
+}


Suggested change

import { parseYaml } from '@redocly/openapi-core';

import { outdent } from 'outdent';

import { collectDocumentMetrics } from './collect-metrics.js';

function yaml(source: string): Record<string, unknown> {

return parseYaml(source) as Record<string, unknown>;

}

import { parseYaml as yaml } from '@redocly/openapi-core';

import { outdent } from 'outdent';

import { collectDocumentMetrics } from './collect-metrics.js';

No need in this wrapper.

tatomyr · 2026-03-16T10:26:41Z

packages/cli/src/commands/score/__tests__/formatters.test.ts

+  workflowDepths: new Map(),
+};
+
+function getStylishOutput(result: ScoreResult = RESULT): string {


It's a bad pattern to use default parameters in tests. They should be as explicit as possible.

feat: add score command

9e40ca1

adamaltman requested review from a team as code owners March 12, 2026 01:21

RomanHotsiy reviewed Mar 12, 2026

View reviewed changes

adamaltman added 2 commits March 11, 2026 20:39

chore: add more tests

2c19efb

adamaltman force-pushed the aa/api-score branch from 79078d6 to 0e328aa Compare March 12, 2026 02:00

adamaltman and others added 3 commits March 11, 2026 21:02

fix: add missing version property to test CommandArgs

ded736c

Made-with: Cursor

chore: add e2e

05bd98c

chore: refactor the visitor to use native pattern

ddfdf10

JLekawa reviewed Mar 12, 2026

View reviewed changes

docs/@v2/commands/score.md Outdated Show resolved Hide resolved

.changeset/add-score-command.md Outdated Show resolved Hide resolved

chore: fix semantic line breaks

1acac45

Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>

tatomyr reviewed Mar 12, 2026

View reviewed changes

adamaltman added 2 commits March 13, 2026 09:56

chore: refactor based on feedback

6cd3af4

fix: adjust walking method and add debugging arguments

92d40f7

tatomyr requested changes Mar 16, 2026

View reviewed changes

Conversation

adamaltman commented Mar 12, 2026

What/Why/How?

Reference

Testing

Screenshots (optional)

Check yourself

Security

Uh oh!

changeset-bot bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

github-actions bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatomyr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatomyr left a comment

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot bot commented Mar 12, 2026 •

edited

Loading

github-actions bot commented Mar 12, 2026 •

edited

Loading

github-actions bot commented Mar 12, 2026 •

edited

Loading