Braintrust eval
ActionsTags
(1)Run Braintrust evals in GitHub Actions and post a live summary comment on the associated pull request.
name: Braintrust evals
on:
pull_request:
push:
permissions:
contents: read
pull-requests: write
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Run evals
uses: braintrustdata/eval-action@v2
with:
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
runtime: nodeImportant
You must specify permissions for the action to leave comments on your PR.
Without these permissions, you'll see GitHub API errors.
| Input | Required | Description |
|---|---|---|
api_key |
Yes | Your Braintrust API key. |
runtime |
Yes | The runtime to use: node, python, or go. |
root |
No | Root directory containing your evals. Defaults to .. |
paths |
No | Paths or glob patterns, relative to root, containing evals to run. Defaults to .. |
package_manager |
No | npm or pnpm for Node; pip or uv for Python; go for Go. Can be omitted for the default package manager. |
use_proxy |
No | Set to true to use the Braintrust proxy at https://braintrustproxy.com/v1, which can cache repetitive LLM calls and speed up evals. Defaults to true. |
terminate_on_failure |
No | Set to true to stop the eval process when an error occurs. Defaults to false. Ignored for Go evals. |
github_token |
No | GitHub token used to create or update PR comments. Defaults to ${{ github.token }}. |
name: Run pnpm evals
on:
pull_request:
push:
# Uncomment to run only when files in the 'evals' directory change.
# paths:
# - "evals/**"
permissions:
contents: read
pull-requests: write
jobs:
eval:
name: Run evals
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Run evals
uses: braintrustdata/eval-action@v2
with:
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
runtime: node
package_manager: pnpm
root: my_eval_dirFor more fully configured workflows, see the examples directory:
-
Node and Python: the action runs
braintrust eval --jsonlfromrootand collects the emitted experiment summaries. -
Go: the action runs
go run ${paths}fromroot. To include Go eval results in the PR comment, print eachExperimentSummaryas one JSON line after callingresult.Summarize(ctx):summary, err := result.Summarize(ctx) if err != nil { log.Fatal(err) } b, err := json.Marshal(summary) if err != nil { log.Fatal(err) } fmt.Println(string(b))
The action creates or updates a single PR comment with a Braintrust link and result table. For example:
| Score | Average | Improvements | Regressions |
|---|---|---|---|
| Levenshtein | 83% (+3pp) | 8 🟢 | 4 🔴 |
| Duration | 1s (0s) | 16 🟢 | 1 🔴 |
Braintrust eval is not certified by GitHub. It is provided by a third-party and is governed by separate terms of service, privacy policy, and support documentation.
Tags
(1)Braintrust eval is not certified by GitHub. It is provided by a third-party and is governed by separate terms of service, privacy policy, and support documentation.