LinkedIn Public Job Scraper

A clean, Docker-free Python project to scrape public job postings from LinkedIn using Playwright. This tool extracts job listings with robust selectors and exports data in multiple formats.

Features

• 🔍 Smart Scraping: Uses Playwright with explicit waits and multiple fallback selectors
• 📊 Multiple Export Formats: CSV, JSON, and SQLite database support
• 🖥️ CLI Interface: Easy-to-use command-line interface
• 🛡️ Robust Parsing: Handles LinkedIn's dynamic page structure with XPath selectors
• 🧪 Test Coverage: Includes unit tests for core functionality
• 📁 Organized Output: Automatic file naming based on search parameters

Quick Start

Prerequisites

• Python 3.9+
• Git

Installation

1. Clone the repository:

   git clone git@github.com:jobpare/linkedIn-scraper.git
   cd linkedIn-scraper

2. Create and activate virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt
   python -m playwright install

Usage

# Basic usage
python cli.py --query "python developer" --location "USA" --output json

# Export to CSV
python cli.py --query "frontend developer" --location "Netherlands" --output csv

# Export to SQLite database
python cli.py --query "data scientist" --location "Canada" --output sqlite

Command Line Options

Option	Required	Description	Choices
--query	Yes	Job search query	Any text
--location	Yes	Job location	Any text
--output	No	Output format	csv, json, sqlite
--help	No	Show help message	-

Project Structure

linkedIn-scraper/
├── scraper/                    # Core scraping logic
│   ├── __init__.py
│   ├── runner.py              # Main scraping orchestration
│   ├── config.py              # URLs, selectors, and parameters
│   ├── parser.py              # HTML parsing with XPath
│   └── exporter.py            # Data export functions
├── tests/                     # Test suite
│   ├── test_runner.py
│   └── test_parser.py
├── data/                      # Output directory (auto-created)
├── cli.py                     # Command-line interface
├── main.py                    # Alternative entry point
├── requirements.txt           # Python dependencies
├── README.md                  # This file
├── LICENSE                    # MIT License
└── .gitignore                # Git ignore rules

Output Data

The scraper extracts the following job information:

• Title: Job position title
• Company: Company name
• Location: Job location
• Link: Direct link to the job posting
• Posted Time: When the job was posted

Output Files

Files are automatically named based on your search parameters:

• data/python developer_USA.json
• data/frontend developer_Netherlands.csv
• data/data scientist_Canada.sqlite

Testing

Run the test suite to verify functionality:

# Run all tests
PYTHONPATH=. pytest tests -v

# Run specific test
PYTHONPATH=. pytest tests/test_parser.py -v

Technical Details

Dependencies

• playwright: Web automation and scraping
• pandas: Data manipulation and CSV/SQLite export
• lxml: HTML parsing with XPath support
• click: CLI framework (optional, using argparse)

Key Features

• Explicit Waits: No manual timeouts, uses Playwright's intelligent waiting
• XPath Selectors: Preferred over CSS for robust element selection
• Multiple Fallbacks: Tries various selectors to handle LinkedIn's dynamic structure
• Error Handling: Graceful degradation and debugging output
• Respectful Scraping: Includes delays and proper user agents

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Setup

# Install development dependencies
pip install pytest

# Run tests
PYTHONPATH=. pytest tests -v

# Test the scraper
python cli.py --query "test" --location "test" --output json

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This tool is for educational and research purposes. Please respect LinkedIn's Terms of Service and robots.txt when using this scraper. Consider implementing rate limiting and respecting website policies in production use.