[Proposal] OpenEval Import/Export Support

[adhabnr-ux]

Overview

We have launched OpenEval (v1.0.0-rc.1), an open standard (Apache 2.0) for portable LLM evaluation datasets. Every eval framework uses incompatible formats — OpenEval provides a standard JSON format + converters + SDKs (TypeScript + Python).

What We're Proposing

Add to_openeval() and from_openeval() methods (~50-100 lines using our SDK).

Resources

• Spec: https://github.com/adhabnr-ux/openeval/blob/main/spec/SPEC.md
• npm SDK: npm install @openeval/sdk
• PyPI SDK: pip install openeval

We would love to collaborate.

[adhabnr-ux]

Update: OpenEval packages published! pip install openeval-sdk. Active responses from Inspect AI, CrewAI, Arize. Would love to collaborate.

[adhabnr-ux]

Correction/update on the package names from my last comment, since they've since changed and I don't want anyone here to hit a broken pip install:

The project renamed from "OpenEval" to EvalPort (repo moved to adhabnr-ux/evalport — old links redirect). The correct install is:

• PyPI: pip install evalport-sdk (imports as openeval, e.g. from openeval.validate import validate_suite)
• npm: npm install evalport-sdk

(pip install openeval-sdk / pip install openeval from my earlier comment don't exist — sorry for the confusion.)

Concrete progress since I opened this: 20 shipped framework adapters (real installable packages under adapters/<name>-openeval-adapter/, each tested against the real validator — Ragas, LangSmith, MLflow, Braintrust, CrewAI, and 15 others are all merged examples if a reference implementation is useful), one integration merged into a third-party repo directly (Inspect AI, PR #4797), and one under active maintainer review right now (TruLens, PR #2697).

If a lighteval to_openeval()/from_openeval() pair is still of interest, happy to sketch a first draft against lighteval's actual result/task objects the way I did for the TruLens and OpenAI-evals PRs above — just say the word, or feel free to open a PR yourselves referencing the spec, no permission needed.