Skip to content

getting-started.md

Latest commit

 

History

History
215 lines (147 loc) · 7.68 KB

getting-started.md

File metadata and controls

215 lines (147 loc) · 7.68 KB
Raw

Getting Started

This guide walks you through installing τ-bench, configuring API keys, and running your first evaluation.

Prerequisites

  • uv — Python package and project manager
  • Python 3.12+ (uv will download it automatically if not present)

Installation

1. Clone the repository

git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench

2. Install τ-bench

uv sync                        # core only (text-mode: airline, retail, telecom, mock)

This creates a virtual environment, installs core dependencies from the lockfile, and enables the tau2 command. The Python version is pinned via .python-version (3.12) — uv will download it automatically if needed.

Optional extras

Install the extras you need:

uv sync --extra voice          # + voice/audio-native features
uv sync --extra knowledge      # + banking_knowledge domain (retrieval pipeline)
uv sync --extra gym            # + gymnasium RL interface
uv sync --extra dev            # + pytest, ruff, pre-commit (required for contributing)
uv sync --extra experiments    # + plotting libs for src/experiments/
uv sync --all-extras           # everything

System dependencies (voice only)

If using voice features (--extra voice), install:

macOS:

brew install portaudio ffmpeg

Note: If you install without -e mode (e.g., uv pip install .), you'll need to set the TAU2_DATA_DIR environment variable to point to your data directory:

export TAU2_DATA_DIR=/path/to/your/tau2-bench/data

3. Verify your installation

uv run tau2 check-data

This checks that your data directory is correctly configured and all required files are present.

Setting Up API Keys

We use LiteLLM to manage LLM APIs, so you can use any LLM provider supported by LiteLLM.

Copy .env.example as .env and edit it to include your API keys:

cp .env.example .env

Voice API Keys (for voice-enabled features)

If you're using voice features, add the following to your .env file:

  • ELEVENLABS_API_KEY — for voice synthesis
  • DEEPGRAM_API_KEY — for voice transcription

Voice Persona Setup (for voice-enabled features)

The voice pipeline uses ElevenLabs voices for the user simulator. The default voice IDs are Sierra-internal and won't work for external users. You need to create your own voices and configure them via environment variables in your .env file:

TAU2_VOICE_ID_MATT_DELANEY=your_voice_id_here
TAU2_VOICE_ID_LISA_BRENNER=your_voice_id_here
# ... (one per persona)

See the Voice Persona Setup Guide for step-by-step instructions on creating matching voices using ElevenLabs Voice Design.

Running Your First Evaluation

Standard text-based evaluation (half-duplex)

tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 \
  --num-trials 1 --num-tasks 5

Results are saved in data/simulations/.

Audio native mode (voice full-duplex)

Prerequisite: Voice mode requires custom ElevenLabs voices for the user simulator. You must set these up before running voice evaluations. See the Voice Persona Setup Guide — the automated script takes care of everything in one command.

tau2 run --domain retail --audio-native --num-tasks 1 --verbose-logs

See the Audio Native Documentation for provider configuration and all options.

Knowledge retrieval evaluation

tau2 run --domain banking_knowledge --retrieval-config bm25 \
  --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-tasks 5

See the Knowledge Retrieval Documentation for retrieval configuration options.

tip: for full agent evaluation that matches the original τ-bench methodology, remove --num-tasks to evaluate on the complete task set (the base split is used by default).

Simulation Output Structure

Results are stored in one of two formats, chosen automatically based on modality:

Text runs — monolithic JSON

Text-based simulations produce a single file containing all data:

data/simulations/<run_name>/
└── results.json             # Metadata, tasks, and all simulation data

Voice runs — directory-based format

Voice simulations contain large tick-level data, so they use a directory-based format that splits simulation data into individual files for efficient checkpointing and streaming:

data/simulations/<run_name>/
├── results.json                        # Metadata and task definitions only
├── simulations/                        # Individual simulation data files
│   ├── sim_0.json
│   ├── sim_1.json
│   └── ...
└── artifacts/                          # Runtime artifacts (with --verbose-logs)
    └── task_<task_id>/
        └── sim_<uuid>/
            ├── sim_status.json         # Simulation status
            ├── task.log                # Per-task log
            ├── audio/
            │   ├── both.wav            # Full conversation audio (stereo)
            │   ├── assistant_labels.txt # Audacity labels for agent speech
            │   ├── user_labels.txt     # Audacity labels for user speech
            │   ├── assistant_tool_calls_labels.txt  # Audacity labels for agent tool calls (when present)
            │   └── user_tool_calls_labels.txt       # Audacity labels for user tool calls (when present)
            └── llm_debug/
                └── *.json              # LLM call logs

Format conversion

You can convert between formats using tau2 convert-results:

# Convert a monolithic JSON to directory format
tau2 convert-results data/simulations/my_run --to dir

# Convert a directory format back to monolithic JSON
tau2 convert-results data/simulations/my_run --to json

Both formats are fully supported by Results.load(), which auto-detects the format on disk.

Viewing Results

tau2 view

This allows you to browse simulation files, view agent performance metrics, inspect individual simulations, and view task details. Works for both standard text and audio native runs.

Configuration

The framework is configured via src/tau2/config.py.

LLM Call Caching

LLM call caching is disabled by default. To enable it:

  1. Install the redis Python package: uv pip install redis
  2. Make sure a Redis server is running
  3. Update the redis config in config.py if necessary
  4. Set LLM_CACHE_ENABLED to True in config.py

Cleanup

To remove all generated files and the virtual environment:

make clean

Next Steps