Skip to content

README.md

Latest commit

 

History

History
302 lines (228 loc) · 8.35 KB

README.md

File metadata and controls

302 lines (228 loc) · 8.35 KB
Raw

🤖 Buddy - AI Virtual Assistant

Your Intelligent Desktop Companion with Gesture Control & Voice Commands

Python OpenCV License Platform Stars

Features • Quick Start • Demo • Documentation • Contributing

📖 About

Buddy is an all-in-one AI virtual assistant that revolutionizes how you interact with your computer. Control your mouse and keyboard with hand gestures, execute commands with your voice, and automate tasks with AI-powered features.

Author: Rajpal Singh Rathore

Features

🎯 Core Components

  1. Virtual Mouse - Hand gesture-based mouse control
  2. Virtual Keyboard - On-screen keyboard with gesture typing
  3. Voice Assistant "Buddy" - AI-powered voice commands

🤖 Assistant Capabilities

  • ✅ Face authentication
  • ✅ Voice command recognition
  • ✅ Spotify integration (play music, search artists)
  • ✅ YouTube control (play videos, search)
  • ✅ WhatsApp automation (messages, calls)
  • ✅ Phone integration (calls, SMS via Phone Link)
  • ✅ Email composition (Gmail)
  • ✅ Web search (Google, product search on 20+ sites)
  • ✅ Note taking (Sticky Notes, file-based)
  • ✅ Reminders (Windows Task Scheduler)
  • ✅ Screenshots
  • ✅ AI chatbot (HuggingChat)
  • ✅ Code generation
  • ✅ Image generation (Stable Diffusion)
  • ✅ Google Maps routes

🖱️ Virtual Mouse Gestures

Gesture Action
Index finger movement Move cursor
Index finger bent + middle straight Left click
Middle finger bent + index straight Right click
Both fingers bent (thumb far) Double click
Both fingers bent (thumb close) Screenshot
Thumb + index touching + move up/down Scroll
Thumb + pinky touching Drawing mode
Thumb + ring finger (1 sec) Close window
Thumb + middle finger (1 sec) Minimize window

🚀 Quick Start

Prerequisites

  • Windows 10/11
  • Python 3.8 or higher
  • Webcam
  • Microphone
  • Internet connection

Installation

  1. Clone or download this repository

    cd C:\VirtualMouseProject

  2. Run setup script

    setup.bat

    This will:

    • Create virtual environment
    • Install all dependencies
    • Verify installation

  3. Configure API keys (Optional) Create a .env file in the project root:

    CLIENT_ID=your_spotify_client_id
CLIENT_SECRET=your_spotify_client_secret
HuggingFaceApiKey=your_huggingface_api_key

  4. Test components

    python test_components.py

  5. Launch the assistant

    start.bat

    Or directly:

    python run.py

📖 Usage

Starting the Assistant

  1. Run start.bat
  2. Complete face authentication when prompted
  3. Wait for "Ready to help" confirmation
  4. Use voice commands or click the mic button
  5. Press Alt + J for quick voice activation

Voice Command Examples

  • "Open YouTube"
  • "Play Despacito on YouTube"
  • "Search for laptop on Amazon"
  • "Send a message to [contact name] on WhatsApp"
  • "Set a reminder for tomorrow at 3 PM to call mom"
  • "Take a screenshot"
  • "Generate an image of a sunset over mountains"
  • "Write a code to sort a list in Python"
  • "What's the weather like?"

Launching Virtual Mouse/Keyboard

  • Voice: "Start virtual mouse" / "Start virtual keyboard"
  • Or use Flask API endpoints (if running app.py)

🛠️ Troubleshooting

Camera not working

  • Check if camera is being used by another application
  • Grant camera permissions to Python

Voice recognition not responding

  • Check microphone permissions
  • Ensure internet connection (uses Google Speech API)
  • Adjust r.pause_threshold in commands.py if needed

Face authentication fails

  • Ensure good lighting
  • Train your face using assist/Engine/auth/sample.py
  • Run assist/Engine/auth/trainer.py to generate trainer.yml

Module not found errors

  • Activate virtual environment: envjarvis\Scripts\activate
  • Reinstall: pip install -r requirements.txt

Spotify not working

📁 Project Structure

VirtualMouseProject/
├── run.py              # Main launcher (multiprocessing)
├── main.py             # Assistant initialization
├── app.py              # Flask API server
├── virtualMouse.py     # Gesture-based mouse
├── virtual_ketboard.py # Gesture-based keyboard
├── requirements.txt    # Dependencies
├── setup.bat           # Installation script
├── start.bat           # Launch script
├── test_components.py  # Component testing
├── assist/
│   ├── Engine/
│   │   ├── commands.py      # Command handler
│   │   ├── features.py      # Feature implementations
│   │   ├── config.py        # Configuration
│   │   ├── db.py            # Database operations
│   │   ├── spotify.py       # Spotify integration
│   │   ├── auth/            # Face authentication
│   │   ├── ImageBot/        # Image generation UI
│   │   └── CodingBuddy/     # Code assistant UI
│   └── www/                 # Web interface
│       ├── index.html
│       ├── main.js
│       └── style.css
└── envjarvis/          # Virtual environment

🎥 Demo

Coming Soon: Full demo video showcasing all features

Screenshots

🎭 Face Authentication

Secure login with facial recognition

🖱️ Virtual Mouse Control

Control cursor with hand gestures

🗣️ Voice Assistant Interface

Beautiful web-based UI with voice commands

🎨 AI Image Generation

Create images from text prompts

Quick Feature Preview

🎵 "Play [song name] on Spotify"
📧 "Send email to [contact]"
🌐 "Search Google for [query]"
🎨 "Generate image of [description]"
💻 "Write code to [task]"

🔧 Development

Adding New Voice Commands

Edit assist/Engine/commands.py and add to allCommands() function

Adding New Contacts

Use the web UI contact form or edit database directly

Training Face Recognition

Run the easy setup script:

.\setup_face_auth.bat

Or manually:

  1. Run python assist/Engine/auth/sample.py to capture face samples
  2. Run python assist/Engine/auth/trainer.py to train the model

🤝 Contributing

Contributions are welcome! Here's how:

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Development Setup

See DEPLOYMENT.md for complete deployment guide.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • OpenCV for computer vision capabilities
  • MediaPipe for hand tracking
  • Eel for Python-JavaScript bridge
  • HuggingFace for AI models
  • All open-source contributors

📧 Contact

Rajpal Singh Rathore

⭐ Star this repo if you find it useful!

Made with ❤️ by Rajpal Singh Rathore

📝 License

This project is open source and available for educational purposes.

🤝 Contributing

Contributions, issues, and feature requests are welcome!

⚠️ Important Notes

  • Some features require API keys (Spotify, HuggingFace)
  • WhatsApp automation may require WhatsApp Desktop app
  • Phone features require Windows Phone Link app
  • Face authentication model needs to be trained with your face

📞 Support

For issues or questions, please create an issue in the repository.