feat: add caching support for Anthropic models via Databricks #6320
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add cache_control fields to requests for Claude models to enable prompt caching: - System message gets cache_control - Last two user messages get cache_control (for incremental caching) - Last tool definition gets cache_control (so all tools are cached as prefix) This reduces costs when using Claude models via Databricks by caching frequently repeated content like system prompts and tool definitions.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds prompt caching support for Claude models accessed through Databricks by implementing Anthropic-style cache_control fields in API requests. This optimization reduces costs by caching frequently repeated content like system prompts, conversation history, and tool definitions.
Key changes:
- Adds
is_claude_model()function to detect Claude models by name - Implements
apply_cache_control_for_claude()to add cache control markers to system messages, the last two user messages, and the last tool definition - Integrates cache control application into the
create_request()flow for Claude models
michaelneale
reviewed
Jan 2, 2026
| "cache_control": { "type": "ephemeral" } | ||
| }]); | ||
| } else if let Some(content_array) = content.as_array_mut() { | ||
| // Content is already an array, add cache_control to the last element |
Collaborator
There was a problem hiding this comment.
probably can drop little comments like this (thanks claude opus!) but others I found helpful (assuming they are true!)
michaelneale
approved these changes
Jan 2, 2026
Collaborator
michaelneale
left a comment
michaelneale
left a comment
There was a problem hiding this comment.
nice (comments helpful - just the odd one liner one) but this is pretty cool, wouldn't really know where to start to work all this out.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add cache_control fields to requests for Claude models to enable prompt caching:
This reduces costs when using Claude models via Databricks by caching frequently repeated content like system prompts and tool definitions.