Google Search Competitor Analysis Tool

A Python-based web scraping tool that performs comprehensive Google search analysis for competitor monitoring across multiple websites and brands.

🚀 Features

• Multi-pattern Search: Uses multiple search strategies for comprehensive coverage
• Date Range Filtering: Search within specific date ranges
• Batch Processing: Efficiently handles large CSV inputs with smart grouping
• Excel Reporting: Generates detailed reports with multiple sheets
• Rate Limiting: Built-in protection against API quota limits
• Error Handling: Robust error handling and recovery mechanisms

📊 What It Does

This tool analyzes competitor mentions across Vietnamese news websites by:

1. Reading keywords, brands, and target websites from a CSV file
2. Performing Google searches with multiple search patterns
3. Collecting and analyzing search results
4. Generating comprehensive Excel reports with detailed analytics

🛠️ Prerequisites

• Python 3.7+
• Google Cloud Console account
• Google Custom Search Engine setup

📦 Installation

1. Clone the repository:

   git clone https://github.com/yourusername/google-search-competitor-analysis.git
   cd google-search-competitor-analysis

2. Install dependencies:

   pip install -r requirements.txt

3. Set up Google API credentials:

   • Go to Google Cloud Console
   • Create a new project or select existing one
   • Enable the Custom Search API
   • Create API credentials (API Key)
   • Set up a Custom Search Engine at Programmable Search Engine

4. Configure the script:

   • Open main.py or main_enhanced.py
   • Replace API_KEY with your Google API key
   • Replace SEARCH_ENGINE_ID with your Custom Search Engine ID

📁 Project Structure

chicom_crawl/
├── main.py                    # Basic version of the scraper
├── main_enhanced.py          # Enhanced version with multiple search patterns
├── input.csv                 # Input data (keywords, brands, websites)
├── requirements.txt          # Python dependencies
├── README.md                # This file
├── LICENSE                  # MIT License
├── .gitignore              # Git ignore file
├── config.py               # Configuration management
├── utils/                  # Utility functions
│   ├── __init__.py
│   ├── search_utils.py     # Search-related utilities
│   └── report_utils.py     # Report generation utilities
└── examples/               # Example files
    ├── sample_input.csv    # Sample input file
    └── sample_output.xlsx  # Sample output file

🚀 Usage

Basic Usage

1. Prepare your input CSV file (input.csv):

   keyword,brand,website
   "Dầu ăn","Tường An",vnexpress.net
   "Dầu ăn","Tường An",dantri.com.vn
   "Gia Vị","Nam Ngư",vnexpress.net

2. Run the basic version:

   python main.py

3. Run the enhanced version (recommended):

   python main_enhanced.py

Configuration

Edit the configuration variables in the script:

# API Configuration
API_KEY = "your_api_key_here"
SEARCH_ENGINE_ID = "your_search_engine_id_here"

# Date Range
START_DATE = "2025-07-01"
END_DATE = "2025-07-31"

# File Names
INPUT_CSV_FILE = "input.csv"
OUTPUT_REPORT_FILE = "competitor_report.xlsx"

📊 Output

The tool generates Excel reports with multiple sheets:

Basic Version (main.py)

• Summary: Results count by brand and keyword
• Raw Data: All search results with details

Enhanced Version (main_enhanced.py)

• Enhanced Summary: Results breakdown by search pattern
• Pattern Analysis: Effectiveness of each search strategy
• Raw Data: Complete results with pattern information

⚠️ Important Notes

API Limitations

• Daily Quota: 10,000 free queries per day
• Rate Limiting: 1-second delay between requests
• API Discontinuation: Current API will be discontinued on January 8, 2025

Migration Required

The current Google Custom Search API will be discontinued. Plan to migrate to Google Vertex AI Search before the deadline.

🔧 Advanced Features

Enhanced Search Patterns

The enhanced version uses multiple search strategies:

1. Full Combination: "keyword" "brand"
2. Brand Only: "brand" (captures brand mentions)
3. Brand with Alternatives: "brand" (related_terms)

Scalability

• Smart Grouping: Reduces API calls by grouping similar searches
• Efficient Processing: Handles 2000+ rows efficiently
• Memory Optimized: Minimal memory usage for large datasets

📈 Performance

Based on analysis with 2000+ row inputs:

• API Calls: ~144 calls for 2000 rows (not 2000 calls!)
• Execution Time: ~2.4 minutes for 2000 rows
• Memory Usage: Minimal, handled efficiently by pandas

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚠️ Disclaimer

This tool is for educational and research purposes. Please ensure compliance with:

• Google's Terms of Service
• Target websites' robots.txt and terms of use
• Applicable data protection regulations

🆘 Support

If you encounter issues:

1. Check the Issues page
2. Ensure your API credentials are correct
3. Verify your input CSV format
4. Check your daily API quota

🔮 Future Enhancements

• Migrate to Google Vertex AI Search
• Add progress tracking for large jobs
• Implement checkpoint/resume functionality
• Add result pagination for more complete data
• Create web interface
• Add automated scheduling
• Implement cost tracking and optimization