🎬 Video Verification Pipeline

Automated Website Verification Agent — A complete pipeline that analyzes screen recordings (MP4) of a user visiting a website to verify which website (domain/URL) was visited and whether a valid certificate or verification screen was shown. The pipeline extracts frames from video, runs OCR on them, and optionally formats results into structured JSON using an LLM.

What Is This?

Project Name

Automated Website Verification Agent

Objective

Build an AI agent that analyzes a screen recording (MP4) of a user visiting a website to verify:

Which website (domain/URL) was visited
Whether a valid certificate or verification screen was shown

The agent outputs a structured JSON result, for example:

{
  "verified": true,
  "domain": "example.com",
  "certificate": {
    "issuer": "Let's Encrypt",
    "valid_from": "2025-01-01",
    "valid_to": "2026-01-01"
  }
}

Problem Definition

Aspect	Description
Input	A screen recording video file (e.g. `.mp4`) showing a user navigating to a target website and displaying the certificate/verification UI.
Output	Confirmation of whether verification was displayed; extracted domain/URL; certificate issuer, subject, validity dates; evidence frames/timestamps; optional confidence score and audit hash.
Challenges	Text in browser UI is small and compressed; each browser differs (Chrome, Edge, Firefox, Safari); video may be blurry; users may upload edited or fake recordings. The system must be accurate, scalable, explainable, and auditable.

Features (Current Implementation)

🎥 Frame Extraction — Extract frames from video at configurable intervals (OpenCV).
🖼️ Image Preprocessing — Grayscale conversion and upscaling for better OCR.
⚡ Parallel OCR Processing — Process all images in parallel for speed (Nanonets or configured OCR).
🤖 AI Formatting — Format OCR results into structured JSON using Groq LLM.
📊 Progress Tracking — Real-time progress for each step.
🛡️ Error Handling — Clear error messages and robust handling.

Quick Start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Copy .env.example to .env and add your API keys
# NANONETS_API_KEY=...
# GROQ_API_KEY=...

# 3. Set VIDEO_PATH in config.py (or use default)

# 4. Run the pipeline
python main.py

The pipeline will: extract frames → run OCR (parallel) → optionally format with LLM.

Folder Structure

video-verification/
├── main.py                 # Main entry point — runs the full pipeline
├── config.py               # All settings: video path, sample rate, OCR, LLM
├── api.py                  # Optional API layer (e.g. FastAPI) for upload/verify
├── pyproject.toml          # Project metadata and dependencies (uv/pip)
├── requirements.txt        # Pip dependencies (if used)
├── .env                    # API keys (create from .env.example)
├── .env.example            # Template for environment variables
├── .gitignore
├── preprocessing/         # Video → frames
│   ├── extract_frame.py   # Extract frames from MP4 (OpenCV)
│   └── gray_scale.py      # Grayscale / preprocessing helpers
├── ocr/                    # OCR integration
│   └── ocr.py             # Single-image OCR (e.g. Nanonets API)
├── extract_data_ocr/       # Batch OCR (parallel over all frames)
│   └── extract_data.py    # Process all frames, aggregate results
├── respose_format/         # LLM formatting of OCR output
│   └── llm_format.py      # Send OCR JSON to Groq, get structured JSON
├── data/
│   ├── input/             # Place input videos here (e.g. test_video.mp4)
│   └── output/            # Extracted frames (e.g. frames02, frames03)
├── results/                # Output JSON files
│   ├── ocr_results*.json       # Raw OCR from all frames
│   └── formatted_ocr_results*.json  # LLM-formatted structured result
└── test/                   # Tests and sample data

Note: Frames folders are auto-numbered (e.g. frames02, frames03). Result filenames use the same number (e.g. ocr_results02.json, formatted_ocr_results02.json).

Local Setup (Step by Step)

1. Prerequisites

Python 3.8+
(Optional) uv for fast dependency management, or use pip with requirements.txt

2. Clone or Download the Project

cd "E:\Projects\video verification"
# or your project path

3. Create and Activate a Virtual Environment

Using uv:

uv venv
.venv\Scripts\activate   # Windows
# source .venv/bin/activate   # Linux/macOS
uv sync

Using pip:

python -m venv .venv
.venv\Scripts\activate   # Windows
pip install -r requirements.txt

4. Environment Variables

Create a .env file in the project root (copy from .env.example):

NANONETS_API_KEY=your_nanonets_api_key_here
GROQ_API_KEY=your_groq_api_key_here

NANONETS_API_KEY — Used for OCR (Nanonets API). Get it from Nanonets.
GROQ_API_KEY — Used for LLM formatting (Groq). Get it from Groq.

5. Configure the Pipeline

Edit config.py:

VIDEO_PATH — Path to your input video (e.g. data/input/test_video.mp4). Put MP4 files in data/input/.
PROCESSING_SETTINGS — sample_rate (e.g. 1/3 = 1 frame per 3 seconds), apply_gray_scale, upscale_factor, max_workers (or None for auto).
LLM_SETTINGS — enabled (True/False), model, max_tokens.

Output paths (frames folder, ocr_results*.json, formatted_ocr_results*.json) are generated automatically from the video path.

6. Run the Pipeline

python main.py

You should see:

Frames extracted to data/output/framesXX/
OCR results written to results/ocr_resultsXX.json
If LLM is enabled, formatted results in results/formatted_ocr_resultsXX.json

Complete Workflow

End-to-end flow of the pipeline:

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐     ┌──────────────────┐
│  Input video    │ ──► │  Extract frames  │ ──► │  OCR (parallel) │ ──► │  LLM formatting  │
│  (data/input/)  │     │  (preprocessing)  │     │  (extract_data)  │     │  (respose_format) │
└─────────────────┘     └──────────────────┘     └─────────────────┘     └──────────────────┘
                                │                          │                         │
                                ▼                          ▼                         ▼
                        data/output/framesXX/       results/ocr_resultsXX.json   results/formatted_ocr_resultsXX.json

Step 1: Video Preprocessing

Validate and read the MP4 (e.g. via OpenCV).
Extract frames at the configured sample rate (e.g. 1 frame per 3 seconds).
Optional: crop to browser address bar / certificate dialog (future improvement).
Apply grayscale and upscaling for better OCR.

Step 2: Text Recognition (OCR)

Run OCR on each extracted frame (via ocr/ocr.py, e.g. Nanonets API).
extract_data_ocr runs all frames in parallel (extract_data.py).
Aggregate text per frame and produce raw OCR JSON for the whole video.

Step 3: Rule-based / Structured Decision (Optional)

In a full verification agent: detect domain across timestamps, look for certificate keywords, compare domain vs certificate subject.
Current code focuses on extraction + LLM formatting; strict verification rules can be added on top of the formatted JSON.

Step 4: LLM Formatting (Optional)

Send aggregated OCR output to a Groq LLM.
Prompt the model to output structured JSON: domain, certificate fields, verified flag, etc.
Write result to results/formatted_ocr_resultsXX.json.

Step 5: Output

ocr_resultsXX.json — Raw OCR per frame.
formatted_ocr_resultsXX.json — Structured verification-style JSON (when LLM step is enabled).

Configuration

All settings are in config.py.

Video Processing

VIDEO_PATH = "data/input/test_video.mp4"

PROCESSING_SETTINGS = {
    "sample_rate": 1/3,       # 1 frame per 3 seconds
    "apply_gray_scale": True,
    "upscale_factor": 1.5,
    "max_workers": None,     # None = auto-detect
}

LLM Formatting

LLM_SETTINGS = {
    "enabled": True,
    "model": "openai/gpt-oss-120b",
    "max_tokens": 7500,
}

To skip the LLM step, set "enabled": False.

Output Files & Schema

Files

File	Description
`results/ocr_resultsXX.json`	Raw OCR results from all frames.
`results/formatted_ocr_resultsXX.json`	Structured JSON from the LLM (domain, certificate, verified, etc.).

Example Output Schema (Verification Result)

{
  "verified": true,
  "confidence": 0.91,
  "domain": "bank.example.com",
  "certificate": {
    "issuer": "DigiCert Inc",
    "subject": "CN=bank.example.com",
    "valid_from": "2025-01-01",
    "valid_to": "2026-01-01"
  },
  "timestamps": [4.0, 5.0, 6.0],
  "video_hash": "sha256:abc123...",
  "evidence": ["frame_004.png", "frame_005.png"],
  "notes": "Domain and certificate fields matched in multiple frames."
}

Exact fields depend on the LLM prompt and post-processing; the pipeline is designed to support this kind of schema.

Solution Approach & Design

The project aligns with a hybrid OCR + optional Vision LLM approach:

Approach	Description	When to Use
OCR-based	Extract frames → OCR (Tesseract/Cloud) → rule-based decision → JSON.	MVPs, on-prem, deterministic, auditable.
Vision LLM	Key frames → vision model (e.g. GPT-4o, Claude) → “Is this a valid certificate?” → aggregate.	Complex UIs, cross-browser; use as assist, not sole source of truth.
Hybrid (recommended)	OpenCV frames → (optional) detector for address bar/cert dialog → OCR for text → rules + optional Vision LLM check → JSON.	Production: accuracy, auditability, scalability.

Current implementation: OCR-based pipeline with optional LLM formatting. Vision models and object detection (e.g. YOLOv8) can be added later for ROI detection and visual validation.

Verification Logic

High-level rule outline for a full verification agent:

URL / domain detection
From OCR text, extract domains (e.g. regex). If the same domain appears in ≥3 distinct timestamps → “consistent domain”.
Certificate presence
Search OCR for keywords: "Certificate", "Issued to", "Issued by", "Valid from", "Valid to". Extract nearby text as structured fields.
Validation decision
If (consistent domain) and (certificate info found) → verified = True, else verified = False. Attach reasons and evidence (e.g. frame paths).

The current code produces OCR + formatted JSON; you can implement these rules in a separate module that consumes formatted_ocr_resultsXX.json.

Implementation Options

Layer	Tool / Model	Purpose	Notes
Video decode	OpenCV	Read and sample frames	Used in `preprocessing/`.
Object detection	YOLOv8 (Ultralytics)	Find address bar / cert window	Optional; not in current repo.
OCR	Nanonets (or Google Vision, AWS Textract, Tesseract)	Extract text	Core for domain & certificate.
Vision LLM	GPT-4o / Gemini 1.5 Pro	Validate visual cues (padlock, popup)	Optional assist.
Logic engine	Python (FastAPI, Pandas)	Decide verified True/False	Deterministic rules.
Storage	S3 / PostgreSQL	Store video, results, hashes	For audit and scaling.

Accuracy & Optimization

Area	Method
Frame sampling	Use 1–2 fps base; optional burst at UI change.
ROI cropping	Detect address bar / popup (e.g. YOLO) and crop before OCR.
Image quality	Upscale, sharpen, grayscale before OCR (already in pipeline).
OCR confidence	Ignore or down-weight low-confidence words.
Temporal consistency	Require same domain across ≥3 frames.
Certificate matching	Compare domain with certificate Subject/SAN.
Model validation	Cross-check Vision model vs OCR where both are used.

Risks and Limitations

Risk	Description	Mitigation
Low video quality	OCR fails on blurred UI	Require minimum resolution (e.g. 1080p+); warn users.
Edited / fake videos	Tampered recordings	Hashing, integrity checks; optional liveness prompts.
OCR misreads	Fonts, compression	Use better OCR or cloud fallback.
LLM hallucination	Wrong text or details	Use OCR as source of truth; LLM for structure/context.
Certificate rotation	Real cert may change later	Record video timestamp; optionally fetch live cert for comparison.

Deployment and Scaling

Language: Python
Framework: FastAPI / Flask (e.g. api.py)
Queue: Celery or AWS SQS for video jobs
Storage: S3 (videos + evidence), PostgreSQL (metadata)
Compute: CPU is enough for current pipeline; GPU optional for YOLO or Vision LLM

Example API:

POST /verify — upload video
Response: JSON verification result

Future Extensions

Liveness token (user must display code on screen).
Automated certificate fetch via TLS for cross-check.
UI template matching for known sites.
Deepfake / splice detection.
Dashboard for reviewing verification results.

Troubleshooting

Issue	What to check
Video not found	Ensure path in `config.py` is correct and file exists under `data/input/`.
OCR API errors	Verify `NANONETS_API_KEY` in `.env`; check internet and API credits.
LLM errors	Verify `GROQ_API_KEY` in `.env`; check rate limits and model name.
Out of memory	Reduce `max_workers` or `sample_rate`; process shorter clips.

Requirements

Python 3.8+
OpenCV (opencv-python or opencv-python-headless)
Requests, python-dotenv
groq (for LLM formatting)
See pyproject.toml or requirements.txt for full list.

Summary

Aspect	Recommendation
Core engine	OCR-based pipeline (frames → OCR → structured output).
Helper	Optional Vision LLM for context and formatting.
Input	MP4 (e.g. H.264, 1080p, 20–30 fps).
Output	Structured JSON with confidence and evidence.
Goal	Accurate, explainable verification that can scale to production.

In short: Use OCR for precision, Vision model for context (optional), and a rule engine for determinism, wrapped in a secure, auditable API.

Made with ❤️ for efficient video verification

Name		Name	Last commit message	Last commit date
extract_data_ocr		extract_data_ocr
ocr		ocr
preprocessing		preprocessing
respose_format		respose_format
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
README.md		README.md
api.py		api.py
config.py		config.py
main.py		main.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation