You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add MLflow AI Gateway section to MLflow integration page (#454)
* docs: add MLflow AI Gateway section to MLflow integration page
Adds a subsection showing how to use OpenAIChatGenerator with MLflow AI
Gateway's OpenAI-compatible endpoint via api_base_url. Includes setup
instructions, standalone and pipeline examples.
* docs: add Create Endpoint screenshot to MLflow AI Gateway section
Copy file name to clipboardExpand all lines: integrations/mlflow.md
+63Lines changed: 63 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,6 +22,8 @@ toc: true
22
22
-[Overview](#overview)
23
23
-[Installation](#installation)
24
24
-[Usage](#usage)
25
+
-[Trace a RAG Pipeline](#trace-a-rag-pipeline)
26
+
-[Use MLflow AI Gateway as an LLM Backend](#use-mlflow-ai-gateway-as-an-llm-backend)
25
27
-[License](#license)
26
28
27
29
## Overview
@@ -130,6 +132,67 @@ Auto-tracing for Haystack can be disabled by calling:
130
132
mlflow.haystack.autolog(disable=True)
131
133
```
132
134
135
+
### Use MLflow AI Gateway as an LLM Backend
136
+
137
+
[MLflow AI Gateway](https://mlflow.org/docs/latest/genai/governance/ai-gateway/) (MLflow ≥ 3.0) is a database-backed LLM proxy that routes requests to multiple providers — OpenAI, Anthropic, Gemini, Mistral, Bedrock, Ollama, and more — through a single OpenAI-compatible endpoint. Provider API keys are stored encrypted on the server, and features like fallback/retry, traffic splitting, and budget tracking are configured in the MLflow UI with no code changes needed.
138
+
139
+
Since the gateway exposes an OpenAI-compatible API, you can use Haystack's built-in `OpenAIChatGenerator` with a custom `api_base_url`:
140
+
141
+
**1. Install MLflow and start the server:**
142
+
143
+
```bash
144
+
pip install mlflow[genai]
145
+
mlflow server --host 127.0.0.1 --port 5000
146
+
```
147
+
148
+
**2. Create a gateway endpoint** in the MLflow UI at `http://localhost:5000`. Navigate to **AI Gateway → Create Endpoint**, select a provider and model, and enter your provider API key. See the [MLflow AI Gateway documentation](https://mlflow.org/docs/latest/genai/governance/ai-gateway/endpoints/) for details.
149
+
150
+

151
+
152
+
**3. Use the endpoint in a Haystack pipeline:**
153
+
154
+
```python
155
+
from haystack import Pipeline
156
+
from haystack.components.builders import ChatPromptBuilder
157
+
from haystack.components.generators.chat import OpenAIChatGenerator
0 commit comments