feat(api): migrate Eval API to FastAPI router (#4345)#4425
feat(api): migrate Eval API to FastAPI router (#4345)#4425leseb merged 10 commits intollamastack:mainfrom
Conversation
meta-cla
bot
added
the
CLA Signed
✱ Stainless preview buildsThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs. ✅ llama-stack-client-node studio · code · diff
❗ llama-stack-client-kotlin studio
✅ llama-stack-client-python studio · code · diff
✅ llama-stack-client-go studio · code · diff
⏳ These are partial results; builds are still running. This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push. |
|
This pull request has merge conflicts that must be resolved before it can be merged. @r-bit-rry please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
r-bit-rry
force-pushed
the
feature/fastapi-router-migration
branch
from
b343911 to
1870c46
Compare
r-bit-rry
requested review from
ashwinb,
bbrowning,
cdoern,
ehhuang,
franciscojavierarceo,
leseb,
mattf and
raghotham
as code owners
| }, | ||
| ) | ||
| async def run_eval( | ||
| benchmark_id_request: Annotated[BenchmarkIdRequest, Depends(get_benchmark_id_request)], |
There was a problem hiding this comment.
Note to reviewers, I wanted to use the Annotated for the entire request but the create_path_dependency function only works with single-field models, JobStatusRequest, JobCancelRequest, and JobResultRequest each have two path parameters (benchmark_id and job_id).
|
@leseb Continuing on our effort here |
|
This pull request has merge conflicts that must be resolved before it can be merged. @r-bit-rry please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
…astack#4425) Migrates the Eval API from `@webmethod` decorators to FastAPI routers, following the pattern established by Batches (llamastack#4191) and Benchmarks (llamastack#4309) migrations. ## Changes - **New module structure**: `src/llama_stack_api/eval/` with `api.py`, `models.py`, `fastapi_routes.py` - **Removed**: Old `src/llama_stack_api/eval.py` with `@webmethod` decorators - **Updated**: Provider implementations to use request models ## Migrated Endpoints | Method | Route | |--------|-------| | POST | `/eval/benchmarks/{benchmark_id}/jobs` | | POST | `/eval/benchmarks/{benchmark_id}/evaluations` | | GET | `/eval/benchmarks/{benchmark_id}/jobs/{job_id}` | | DELETE | `/eval/benchmarks/{benchmark_id}/jobs/{job_id}` | | GET | `/eval/benchmarks/{benchmark_id}/jobs/{job_id}/result` | ## Testing - All pre-commit hooks pass - OpenAPI specs regenerated - Type checking verified Closes llamastack#4345
…4683) # What does this PR do? Adds a backward compatibility layer for the Eval API to support both old-style (individual parameters) and new-style (request objects introduced in #4425) calling conventions. When using the deprecated old-style, a `DeprecationWarning` is emitted. **Changes:** - Add `compat.py` module with resolver functions - Update `EvalRouter` to accept both calling styles - Export compat helpers from `llama_stack_api` **Example:** #### New style (preferred, introduced in #4425) `await eval.run_eval(RunEvalRequest(benchmark_id="...", benchmark_config=...))` #### Old style (deprecated, emits warning - added in this PR) `await eval.run_eval(benchmark_id="...", benchmark_config=...)` ## Test Plan <!-- Describe the tests you ran to verify your changes with result summaries. *Provide clear instructions so the plan can be easily re-executed.* --> - Run unit tests for current changes: `uv run pytest tests/backward_compat/test_eval_compat.py -v` - Make sure #4425 tests pass - `uv run pytest tests/unit/test_eval_models.py tests/unit/providers/nvidia/test_eval.py` - Run pre-commit hooks: `uv run pre-commit run --all-files` - Verify deprecation warnings are emitted when using old-style parameters cc @leseb @r-bit-rry
3 participants
Migrates the Eval API from
@webmethoddecorators to FastAPI routers, following the pattern established by Batches (#4191) and Benchmarks (#4309) migrations.Changes
src/llama_stack_api/eval/withapi.py,models.py,fastapi_routes.pysrc/llama_stack_api/eval.pywith@webmethoddecoratorsMigrated Endpoints
/eval/benchmarks/{benchmark_id}/jobs/eval/benchmarks/{benchmark_id}/evaluations/eval/benchmarks/{benchmark_id}/jobs/{job_id}/eval/benchmarks/{benchmark_id}/jobs/{job_id}/eval/benchmarks/{benchmark_id}/jobs/{job_id}/resultTesting
Closes #4345