feat: Add /v1/responses support to the gateway

[brightsparc]

Fixes #991

Description

This PR adds OpenAI-compatible /v1/responses support to the gateway.

The SDK already supports the Responses API through responses(...) / aresponses(...), but the gateway only exposed /v1/chat/completions and /v1/messages. That meant clients using Responses-specific functionality had to bypass the gateway entirely, which breaks centralized auth, rate limiting, budgeting, logging, and observability.

This change adds a new POST /v1/responses route that:

• reuses the existing async aresponses(...) implementation
• stays consistent with the existing gateway streaming pattern
• only allows providers where SUPPORTS_RESPONSES=True
• preserves Responses-only request fields such as reasoning, previous_response_id, and include=["reasoning.encrypted_content"]
• supports both non-streaming and streaming Responses traffic

The rationale for adding this route is that the Responses API is not just a rename of chat completions. It enables additional capabilities, including encrypted reasoning passthrough via include=["reasoning.encrypted_content"], which are not available via the chat completions gateway route.

PR Type

• 🆕 New Feature

Relevant issues

Closes the manually-filed issue for adding /v1/responses support to the gateway.

Checklist

• I understand the code I am submitting.
• I have added unit tests that prove my fix/feature works
• I have run this code locally and verified it fixes the issue.
• New and existing tests pass locally
• Documentation was updated where necessary
• I have read and followed the contribution guidelines
• AI Usage:
  • No AI was used.
  • AI was used for drafting/refactoring.
  • This is fully AI-generated.

AI Usage Information

• AI Model used: GPT-5
• AI Developer Tool used: Codex
• Any other info you'd like to share: AI was used to draft and implement the gateway route, tests, and PR copy. The change was then reviewed and validated locally with gateway-specific pytest coverage and linting.

When answering questions by the reviewer, please respond yourself, do not copy/paste the reviewer comments into an AI system and paste back its answer. We want to discuss with you, not your AI :)

• I am an AI Agent filling out this form (check box if true)

[codecov]

Codecov Report

❌ Patch coverage is 94.59459% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/any_llm/gateway/api/routes/responses.py	94.36%	3 Missing and 1 partial ⚠️

Files with missing lines	Coverage Δ
src/any_llm/gateway/api/main.py	100.00% <100.00%> (ø)
src/any_llm/gateway/streaming.py	100.00% <100.00%> (+56.41%)	⬆️
src/any_llm/gateway/api/routes/responses.py	94.36% <94.36%> (ø)

... and 60 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[njbrake]

Note: This review was generated by Claude after discussion with @njbrake.

PR #992 Review: feat: Add /v1/responses support to the gateway

Summary

This PR adds a new POST /v1/responses route to the FastAPI gateway, enabling OpenAI Responses API passthrough. It includes a new route file (responses.py) following the same patterns as chat.py and messages.py, a RESPONSES_STREAM_FORMAT in the shared streaming module, router registration, and test coverage for basic completion, auth, unsupported providers, streaming, and reasoning field passthrough.

Code Review

File	Assessment
src/any_llm/gateway/api/main.py	Clean. Import added, router registered. No issues.
src/any_llm/gateway/streaming.py	Clean. New RESPONSES_STREAM_FORMAT follows existing patterns correctly. yield_done_on_error=False matches the Responses API SSE spec.
src/any_llm/gateway/api/routes/responses.py	Generally solid, follows the messages.py pattern well. Several issues noted below.
tests/gateway/test_responses_endpoint.py	Decent coverage. A few structural issues and missing cases.

Issues Found

1. data: [DONE] sentinel may not match the Responses API spec

streaming.py:32 sets done_marker="data: [DONE]\n\n". The OpenAI Responses API streaming spec does NOT send a data: [DONE] sentinel. The Responses API uses named SSE events (event: response.completed, etc.) and simply closes the stream. Sending data: [DONE] is a Chat Completions convention that could confuse clients using the Responses API. This should be verified against the actual OpenAI spec and corrected if needed.

2. _usage_to_completion_usage uses getattr instead of direct attribute access

responses.py:36-38 uses getattr(usage, "input_tokens", None) etc. Per project guidelines: "Prefer direct attribute access over getattr when the field is typed." The usage parameter is typed Any, so this is borderline acceptable, but the function is only ever called on ResponseResource.usage / Response.usage which have typed fields. Consider narrowing the type or documenting why getattr is needed.

3. Coverage gap: 82.5% patch coverage (14 lines missing)

The Codecov report flags responses.py at 81.81%, below the ~85% target. Missing coverage likely includes the error paths. Tests should be added for:

• Provider call raising an exception (non-streaming)
• Provider call raising an exception (streaming)
• Non-streaming with no usage data returned
• The _usage_to_completion_usage edge cases (all None fields)

4. No test for API key auth (non-master-key)

The messages endpoint tests include test_messages_endpoint_bearer_auth and test_messages_endpoint_x_api_key_header. The responses tests only test master key auth. Should add a test with a regular API key.

5. ConfigDict(extra="allow") on ResponsesRequest is undocumented

The chat and messages request models don't use extra="allow". This is correct for passthrough of Responses-only fields (reasoning, include, etc.), but a brief comment explaining why would help future readers.

Compliance with Project Guidelines

• Naming conventions: Follows existing patterns (router prefix, tags, etc.) ✅
• Type hints: Present throughout. Uses Any in a few places but consistent with codebase style ✅
• Test structure: Standalone functions, no class-based grouping, imports at top ✅
• Commit format: feat: Add /v1/responses support to the gateway follows Conventional Commits ✅
• getattr usage: See issue Add an acompletion method to api #2 above ⚠️
• No emdashes: Clean ✅

Suggestions (non-blocking)

1. Consider narrowing _usage_to_completion_usage to accept the specific OpenAI ResponseUsage type instead of Any for better type checking.
2. The request_fields["input_data"] = request_fields.pop("input") rename at line 101 is clear but could use a brief comment explaining the mapping from OpenAI's input to the SDK's input_data parameter name.

Verdict

Request Changes

The PR is well-structured and follows existing gateway patterns closely. Two items to resolve before merging:

1. Blocking: The data: [DONE] done marker in RESPONSES_STREAM_FORMAT may not match the Responses API streaming protocol. Please verify against the OpenAI spec and correct if needed.
2. Blocking: Coverage is at 82.5% (target ~85%). Add error-path tests for provider failures in both streaming and non-streaming modes, and a basic API key auth test.

[njbrake]

Hi @brightsparc , Could you follow this guide to allow maintainers of any-llm to push to your fork branch: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork?

This would help us avoid the back and forth to let me help get the PRs over the finish line slightly faster

[brightsparc]

Hi @brightsparc , Could you follow this guide to allow maintainers of any-llm to push to your fork branch: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork?

This would help us avoid the back and forth to let me help get the PRs over the finish line slightly faster

Hi @njbrake I added you as a maintainer on the fork, because I forked into via an organization repo I can't add any mnaintainer, but can add others if required. For next time I can fork via my personal profile.

[brightsparc]

I’ve also updated this from base banch. Let me know if there are any other changes you would like to see @njbrake

[tbille]

Hey @brightsparc

First, thanks so much for the contributions, and sorry for the late answers.
We are currently splitting the gateway into its own repository: https://github.com/mozilla-ai/gateway

We would love to bring your work to this new codebase. If you are ok, we can merge your pull requests, and I will bring over those features on the new gateway.

Unfortunately, I will not be able to bring directly your commits with your name in the new repository.

If this is an issue for you, I can suggest that you make those PRs against the new repository. If not, then I will merge your PR, and I will migrate your code.

The timing made it seem like everything happened at once 😅

[brightsparc]

Hey @brightsparc

First, thanks so much for the contributions, and sorry for the late answers. We are currently splitting the gateway into its own repository: https://github.com/mozilla-ai/gateway

We would love to bring your work to this new codebase. If you are ok, we can merge your pull requests, and I will bring over those features on the new gateway.

Unfortunately, I will not be able to bring directly your commits with your name in the new repository.

If this is an issue for you, I can suggest that you make those PRs against the new repository. If not, then I will merge your PR, and I will migrate your code.

The timing made it seem like everything happened at once 😅

Hi @njbrake I am not precious about need to own this contribution, but also happy to create a PR and land if they are not yet in the repo.

Perhaps you could just include in the comment like co-wrote with @brightsparc similar to what Claude does.

[tbille]

perfect. I will merge those PRs and make a last gateway release so you can get those nice contributions. I will then move over all those features to the new repo (with you as co-authored 😄 )
Once your tests are passing I will 👍 it

[tbille]

@brightsparc if you don't want to bother fixing the tests I will just close your PRs since they were all migrated to the new gateway.

Let me know if you migrated or if you want those to be merged. These PRs will be the last ones before I will close all PRs against this gateway.

[brightsparc]

Yes I realized. Fine to close.