french-ecommerce-lead-scraper

🇫🇷 Enterprise French E-Commerce & B2B Lead Scraping Engine

Asynchronous, resilient, CRM-ready data extraction for French e-commerce & B2B intelligence

---

📋 Overview

This engine extracts structured product and B2B lead data from French e-commerce platforms — pricing, availability, brand, and EAN/GTIN reference data — and turns it into a clean, deduplicated, CRM-ready dataset (CSV or Excel).

It is built around a strict separation of concerns:

Layer	Responsibility	Module
Extraction	Async browser automation, anti-bot resilience, HTML retrieval	scraper/engine.py
Parsing	DOM → structured, typed records	scraper/engine.py (BeautifulSoup)
Transformation	Currency/EAN cleaning, dedup, CRM schema mapping	scraper/cleaner.py
Orchestration	CLI, logging, pipeline wiring	main.py

✨ Key Features

• 🛡️ Anti-Bot Resilience — per-context User-Agent rotation, French locale/timezone fingerprinting, randomized human-like pacing, and automation-flag suppression to reduce detection surface.
• 🔄 Proxy Rotation — round-robin HTTP(S) proxy pool with credential support, configurable per run.
• ⚡ Fully Asynchronous — built on asyncio + Playwright, with semaphore-bounded concurrency for predictable throughput.
• 🔁 Retry & Backoff — exponential backoff with jitter on timeouts and rate-limit/bot-detection responses (HTTP 403/429/503).
• 🧹 Enterprise Data Cleaning — French currency parsing (1 299,99 € → 1299.99), EAN-13 checksum validation, Unicode NFC normalization for accented text, and deduplication.
• 📊 CRM-Ready Exports — CSV (UTF-8 BOM for Excel/French accents) and formatted Excel, with column mapping to a standard CRM import schema.
• 📝 Structured Logging — colorized console output + rotating file logs via loguru.
• ✅ Typed & Validated — Pydantic models throughout for config, site profiles, and output records.

🏗️ Architecture

flowchart LR
    A["Target URLs"] --> B["EnterpriseScraper<br/>(async context manager)"]
    B --> C["Browser Context Pool"]
    C --> D["UA + Locale Rotation"]
    C --> E["Proxy Rotation"]
    B --> F["Retry + Backoff Handler"]
    F --> G["Raw HTML"]
    G --> H["BeautifulSoup Parser<br/>+ SiteProfile Selectors"]
    H --> I["ProductRecord<br/>(Pydantic Model)"]
    I --> J["DataCleaner"]
    J --> K["Currency Normalization"]
    J --> L["EAN-13 Validation"]
    J --> M["Deduplication"]
    J --> N["CRM Export<br/>(CSV / Excel)"]

    style B fill:#2563eb,color:#fff
    style J fill:#059669,color:#fff
    style N fill:#d97706,color:#fff

Loading

📁 Project Structure

french-ecommerce-scraper/
├── main.py                 # CLI entry point (scrape / clean / pipeline)
├── requirements.txt
├── LICENSE
├── scraper/
│   ├── __init__.py
│   ├── engine.py            # EnterpriseScraper — async extraction
│   └── cleaner.py           # DataCleaner — pandas cleaning pipeline
├── examples/
│   ├── urls.txt              # Example target-URL file
│   └── proxies.txt           # Example proxy-pool file
├── data/                    # Runtime output (git-ignored)
└── logs/                    # Rotating log files (git-ignored)

🚀 Installation

git clone https://github.com/<your-username>/french-ecommerce-scraper.git
cd french-ecommerce-scraper

python3.11 -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

pip install -r requirements.txt
playwright install chromium     # downloads the browser binary

▶️ Usage

1. Scrape only — extract raw product data to CSV:

python main.py scrape \
    --urls-file examples/urls.txt \
    --site-profile sanitino_fr \
    --output data/raw_products.csv \
    --concurrency 8

2. Clean only — run the CRM-ready pipeline on an existing raw file:

python main.py clean \
    --input data/raw_products.csv \
    --output data/leads_final.xlsx \
    --format excel

3. Full pipeline — scrape + clean in a single command:

python main.py pipeline \
    --urls-file examples/urls.txt \
    --site-profile sanitino_fr \
    --output data/leads_final.xlsx \
    --proxies-file examples/proxies.txt

Run python main.py <command> --help for the full flag reference.

⚙️ Configuration

Runtime behaviour is controlled via ScraperSettings (scraper/engine.py) — concurrency, retry count, timeouts, delay bounds, proxies, and the User-Agent pool are all adjustable without touching engine code.

Site-specific extraction is controlled via SiteProfile — a name, base URL, and a dict of CSS selectors per field (product_name, price, ean, brand, availability). Two example profiles are included (sanitino_fr, hansgrohe_fr).

Important: the bundled selectors are illustrative placeholders. Real e-commerce markup varies by template/category and changes over time — inspect the live DOM (browser DevTools → Elements) and update SITE_PROFILES before pointing this engine at a production target.

⚖️ Compliance & Responsible Use

This project is designed with production compliance in mind, not just technical throughput:

• Rate-aware by design — concurrency, delay, and retry ceilings are all first-class, tunable settings, so the engine can run at a "polite crawler" pace rather than a maximal one.
• robots.txt & ToS — always review and comply with a target site's crawling policy and Terms of Service before running against it in production; this repository is a technical showcase, not a license to scrape any given site.
• GDPR-aware for B2B data — scraped B2B contact data can constitute personal data under GDPR. Pair this pipeline with a documented legal basis (e.g. legitimate interest), data-minimization practices, and a retention policy before storing or contacting leads.

🗺️ Roadmap

• pytest + pytest-asyncio test suite (selector parsing, EAN/currency edge cases)
• pyproject.toml with ruff + mypy strict config
• GitHub Actions CI (lint, type-check, test on PR)
• Pluggable storage backends (Postgres / BigQuery) alongside CSV/Excel
• Optional playwright-stealth integration for higher-fingerprint-risk targets

📄 License

Released under the MIT License.