Skip to content

tensorzero_setup.md

Latest commit

 

History

History
84 lines (65 loc) · 2.29 KB

tensorzero_setup.md

File metadata and controls

84 lines (65 loc) · 2.29 KB

TensorZero Gateway Configuration Guide

Overview

TensorZero provides a unified gateway for connecting to various LLM providers. It handles authentication, rate limiting, and model management.

For official TensorZero documentation, see:
TensorZero Documentation

Configuration Steps

  1. Edit the configuration file at tensorzero_config/tensorzero.toml:
# Define a function for your LLM tasks
[functions.generate_haiku]
type = "chat"  # The type of task

# Define model variants for this function
[functions.generate_haiku.variants.default]
type = "chat_completion"
model = "openai::gpt-4o-mini"  # Format: provider::model-name

  1. Supported Model Providers:

    • OpenAI (format: openai::model-name)
    • Anthropic (format: anthropic::model-name)
    • Google (format: google::model-name)
    • Local models (format: local::model-name)

  2. Defining Multiple Variants:

[functions.generate_response.variants.fast]
type = "chat_completion"
model = "anthropic::claude-instant"
max_tokens = 256

[functions.generate_response.variants.quality]
type = "chat_completion" 
model = "anthropic::claude-2"
max_tokens = 1024
  1. Additional Configuration Options:
    • temperature: Controls randomness (0-1)
    • max_tokens: Maximum response length
    • top_p: Nucleus sampling threshold

API Key Configuration in Docker

Set your provider API keys in the Docker Compose environment:

# docker-compose-tensorzero.yml
services:
  tensorzero:
    environment:
      # For OpenAI
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      
      # For Anthropic  
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      
      # For Google
      GOOGLE_API_KEY: ${GOOGLE_API_KEY}

Add these to your .env file:

# .env
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key  
GOOGLE_API_KEY=your_google_key

Migrating from Direct LLM Configuration

  1. Remove old LLM settings from:

    • config.py (LLM_MODEL_TYPE, LLM_MODEL, LLM_API_URL)
    • secrets.yaml (llm_api_key)

  2. Convert your existing configuration to TensorZero format:

    • Old: LLM_MODEL_TYPE = "openai" → New: model = "openai::gpt-4"
    • Old: LLM_MODEL_TYPE = "anthropic" → New: model = "anthropic::claude-2"

  3. Test your new configuration with existing workflows