Voice Chat AI - Advanced Conversational Platform

A sophisticated real-time conversational AI platform with multi-provider support, emotional intelligence, and advanced conversation features.

🌟 Features

Core Capabilities

• Multi-Provider AI Integration: OpenAI, Ollama, XAI, and Anthropic support with automatic failover
• Advanced TTS System: Multiple TTS providers (OpenAI, ElevenLabs, XTTS, Kokoro) with emotional voice modulation
• Character Management: Dynamic character system with mood-based prompt adaptation
• Real-time Communication: WebSocket support with heartbeat monitoring and state synchronization
• Sentiment Analysis: Intelligent mood detection and character response adaptation
• Enhanced Conversations: Personality adaptation, topic tracking, and conversation analytics

Advanced Features

• Emotional Intelligence: Sentiment analysis with mood tracking and adaptive responses
• Dynamic Personality: Real-time personality adaptation based on conversation context
• Emotional TTS: Voice modulation based on detected emotions and conversation mood
• Topic Tracking: Intelligent conversation flow analysis and topic management
• State Management: Comprehensive state synchronization across sessions
• Configuration Management: Flexible, multi-source configuration system

🚀 Quick Start

Prerequisites

• Python 3.11 or higher
• Node.js 16+ (for frontend components)
• Redis (optional, for enhanced caching)
• PostgreSQL or SQLite (for data persistence)

Installation

1. Clone the Repository

git clone https://github.com/your-org/voice-chat-ai.git
cd voice-chat-ai

2. Install Dependencies

pip install -r requirements.txt

3. Environment Configuration

cp .env.example .env
# Edit .env with your API keys and configuration

4. Initialize Database

python -m app.db.init

5. Run the Application

python -m app.main_new

Docker Quick Start

docker-compose up -d

📋 Configuration

Environment Variables

AI Providers

# OpenAI Configuration
OPENAI_API_KEY=your_openai_api_key
OPENAI_MODEL=gpt-4o-mini
OPENAI_BASE_URL=https://api.openai.com/v1

# Anthropic Configuration
ANTHROPIC_API_KEY=your_anthropic_api_key
ANTHROPIC_MODEL=claude-3-sonnet-20240229

# XAI Configuration
XAI_API_KEY=your_xai_api_key
XAI_MODEL=grok-beta

# Primary Provider
MODEL_PROVIDER=openai

TTS Providers

# ElevenLabs Configuration
ELEVENLABS_API_KEY=your_elevenlabs_api_key

# TTS Provider Selection
TTS_PROVIDER=openai

Server Configuration

HOST=0.0.0.0
PORT=8000
DEBUG=false
ENVIRONMENT=production

Database Configuration

DATABASE_URL=postgresql://user:password@localhost:5432/voicechat
REDIS_URL=redis://localhost:6379

Security Configuration

SECRET_KEY=your-secret-key-min-32-characters

Configuration File

Create config.yaml for advanced configuration:

ai_providers:
  primary_provider: "openai"
  fallback_providers: ["ollama", "xai", "anthropic"]
  openai_model: "gpt-4o-mini"
  openai_temperature: 0.8

tts_providers:
  primary_provider: "openai"
  fallback_providers: ["elevenlabs"]
  openai_voice: "alloy"

features:
  sentiment_analysis: true
  enhanced_conversations: true
  emotional_tts: true
  personality_adaptation: true
  topic_tracking: true
  websocket_support: true
  analytics: true

server:
  host: "0.0.0.0"
  port: 8000
  workers: 4
  log_level: "info"

security:
  rate_limit_requests_per_minute: 60
  rate_limit_requests_per_hour: 1000
  cors_origins: ["*"]

🏗️ Architecture

System Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend      │    │   WebSocket     │    │   API Gateway   │
│   (React/Vue)   │◄──►│   Manager       │◄──►│   (FastAPI)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                       ┌─────────────────────────────────┼─────────────────────────────────┐
                       │                                 │                                 │
                ┌─────────────┐                   ┌─────────────┐                 ┌─────────────┐
                │ AI Provider │                   │ TTS Provider│                 │ Character   │
                │ Service     │                   │ Service     │                 │ Service     │
                └─────────────┘                   └─────────────┘                 └─────────────┘
                       │                                 │                                 │
        ┌─────────────────────────────┐    ┌─────────────────────────────┐    ┌─────────────────────────────┐
        │ OpenAI │ Ollama │ XAI │ ... │    │ OpenAI │ ElevenLabs │ ... │    │ Sentiment │ Enhanced │ ... │
        └─────────────────────────────┘    └─────────────────────────────┘    └─────────────────────────────┘

Core Components

Service Layer

• AIProviderService: Multi-provider AI integration with failover
• TTSProviderService: Text-to-speech with emotional modulation
• CharacterService: Character management and prompt adaptation
• SentimentAnalysisService: Mood detection and emotional intelligence
• EnhancedConversationService: Advanced conversation features
• StateManager: Distributed state management
• ConfigurationManager: Multi-source configuration management

API Layer

• Character Management: CRUD operations for characters
• Conversation API: Enhanced conversation endpoints
• Provider Management: AI and TTS provider configuration
• Sentiment Analysis: Mood detection and adaptation endpoints
• Configuration API: Runtime configuration management
• WebSocket API: Real-time communication

Data Models

• Character Models: Character definitions and metadata
• Conversation Models: Message history and context
• Sentiment Models: Emotion and mood tracking
• Audio Models: TTS and STT configurations
• State Models: Application state management

🎭 Character System

Character Definition

Characters are defined in JSON files with the following structure:

{
  "name": "assistant",
  "display_name": "AI Assistant",
  "description": "A helpful AI assistant",
  "personality": "Friendly and knowledgeable",
  "background": "Professional AI assistant",
  "speaking_style": "Clear and helpful",
  "system_prompt": "You are a helpful AI assistant...",
  "example_messages": [
    {
      "role": "user",
      "content": "Hello!"
    },
    {
      "role": "assistant", 
      "content": "Hello! How can I help you today?"
    }
  ],
  "voice_config": {
    "provider": "openai",
    "voice_id": "alloy",
    "speed": 1.0,
    "pitch": 1.0
  },
  "mood_prompts": {
    "happy": "Respond with enthusiasm and joy",
    "sad": "Respond with empathy and comfort",
    "excited": "Match their excitement level",
    "calm": "Maintain a peaceful, centered tone"
  }
}

Creating Custom Characters

1. Create Character File

cp app/characters/assistant.json app/characters/my_character.json

2. Edit Character Properties

{
  "name": "my_character",
  "display_name": "My Custom Character",
  "system_prompt": "You are my custom character with unique traits...",
  "voice_config": {
    "provider": "elevenlabs",
    "voice_id": "custom_voice"
  }
}

3. Load Character via API

curl -X POST "http://localhost:8000/characters/" \
  -H "Content-Type: application/json" \
  -d @app/characters/my_character.json

🔊 Audio System

TTS Configuration

OpenAI TTS

tts_config = {
    "provider": "openai",
    "voice_id": "alloy",  # alloy, echo, fable, onyx, nova, shimmer
    "model": "tts-1-hd",
    "speed": 1.0
}

ElevenLabs TTS

tts_config = {
    "provider": "elevenlabs",
    "voice_id": "your_voice_id",
    "stability": 0.5,
    "similarity_boost": 0.5
}

Emotional TTS

emotional_config = {
    "base_voice": "alloy",
    "emotional_intensity": "moderate",
    "voice_style": "empathetic",
    "speed_adjustment": 1.0,
    "pitch_adjustment": 1.0
}

🧠 Sentiment Analysis

Emotion Detection

The system automatically detects:

• Primary Emotions: Happy, sad, angry, fearful, surprised, neutral
• Mood States: Excited, calm, melancholy, anxious, confident, playful
• Sentiment Scores: Positive, negative, neutral, compound

Mood Adaptation

Characters automatically adapt responses based on:

• User's detected emotional state
• Conversation history and patterns
• Character personality configuration
• Contextual conversation flow

API Usage

# Analyze sentiment
response = await sentiment_analyzer.analyze_sentiment("I'm feeling great today!")

# Adapt character response
adapted_prompt = await mood_adaptation_service.adapt_character_response(
    character=character,
    user_message="I'm feeling great today!",
    sentiment_analysis=response,
    conversation_id="conv_123",
    user_id="user_456"
)

🚀 Deployment

Production Deployment

Using Docker

# Build image
docker build -t voice-chat-ai .

# Run container
docker run -d \
  --name voice-chat-ai \
  -p 8000:8000 \
  -e OPENAI_API_KEY=your_key \
  -e DATABASE_URL=your_db_url \
  voice-chat-ai

Using Docker Compose

version: '3.8'
services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - DATABASE_URL=postgresql://user:pass@db:5432/voicechat
    depends_on:
      - db
      - redis

  db:
    image: postgres:15
    environment:
      POSTGRES_DB: voicechat
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:

Manual Deployment

# Install dependencies
pip install -r requirements.txt

# Set environment variables
export OPENAI_API_KEY=your_key
export DATABASE_URL=your_db_url
export ENVIRONMENT=production

# Run with Gunicorn
gunicorn app.main_new:app -w 4 -k uvicorn.workers.UvicornWorker

Scaling Considerations

Horizontal Scaling

• Use load balancer (nginx, HAProxy)
• Redis for session storage
• PostgreSQL with read replicas
• Container orchestration (Kubernetes, Docker Swarm)

Performance Optimization

• Enable Redis caching
• Use CDN for static assets
• Configure database connection pooling
• Implement response caching
• Use async/await patterns

📊 Monitoring and Logging

Health Checks

• /health - Application health
• /config/health - Configuration system health
• /sentiment/health - Sentiment analysis health
• /enhanced/health - Enhanced features health

Metrics

• Request/response times
• Provider response times
• Error rates by endpoint
• WebSocket connection counts
• Sentiment analysis accuracy
• Character usage statistics

Logging Configuration

logging:
  level: "INFO"
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  file_path: "/var/log/voice-chat-ai.log"
  max_file_size: 10485760  # 10MB
  backup_count: 5

🔧 Development

Development Setup

# Install development dependencies
pip install -r requirements-dev.txt

# Run in development mode
python -m app.main_new --reload

# Run tests
pytest

# Code formatting
black app/
isort app/

# Type checking
mypy app/

API Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Testing

# Run all tests
pytest

# Run specific test categories
pytest -m unit
pytest -m integration

# Run with coverage
pytest --cov=app --cov-report=html

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

Code Style

• Follow PEP 8
• Use type hints
• Write comprehensive docstrings
• Add unit tests for new features

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Documentation: Full Documentation
• Issues: GitHub Issues
• Discussions: GitHub Discussions

🔗 Related Projects

• OpenAI API
• Anthropic Claude
• ElevenLabs
• FastAPI
• WebRTC