Page Pulse

Page Pulse is a full-stack web application for analyzing individual web pages. Submit a URL, and the app fetches the page, extracts structure and link metrics, checks link accessibility, and presents the results in a Material UI dashboard.

Stack: Go (Gin) API · React / TypeScript · MySQL · Docker Compose

Features

• Page analysis — title, HTML version (doctype-based), heading counts (H1–H6), login-form detection
• Link analysis — internal vs external link counts, broken-link detection (HTTP HEAD), broken-link list in the detail view
• URL management — add, start, stop, delete, and re-analyze URLs from the dashboard
• Batch operations — start, stop, delete, or re-run analysis for multiple URLs at once
• Live status — dashboard and detail views poll while a crawl is running
• Cancellable crawls — stop requests cancel in-flight HTTP work; only one crawl runs per URL at a time
• Security — SSRF protection (blocks private/loopback targets), per-IP rate limits on login and crawl triggers
• Authentication — bearer-token sessions (default admin / admin for local use)

Analysis report

For each URL, the app produces a report with:

Section	Details
Basic info	URL, page title, status, HTML version, crawl timestamp, login form yes/no
Link analysis	Internal / external / broken counts, breakdown bars, list of broken links
Header tags	Count of H1–H6 elements

Crawls are single-page (the exact URL submitted). The app does not spider the whole site.

---

Quick start (Docker Compose)

1. Clone and enter the project:

   git clone <your-repo-url>
   cd page-pulse

2. Start all services:

   docker-compose up --build

   Service	Port
   Frontend	5173
   Backend API	8080
   MySQL	3306

3. Open the app: http://localhost:5173

4. Log in: username admin, password admin

5. Health check: http://localhost:8080/health

The frontend proxies /api to the backend container (page-pulse-backend).

---

Manual local development

1. MySQL

docker run --name page-pulse-db \
  -e MYSQL_ROOT_PASSWORD=rootpass \
  -e MYSQL_DATABASE=page_pulse \
  -e MYSQL_USER=page_pulse_user \
  -e MYSQL_PASSWORD=page_pulse_pass \
  -p 3306:3306 -d mysql:8.0

2. Backend

cd backend
go run main.go

Variable	Default	Description
DB_HOST	localhost	MySQL host
DB_PORT	3306	MySQL port
DB_USER	root	MySQL user
DB_PASS	(empty)	MySQL password
DB_NAME	page_pulse	Database name
ENVIRONMENT	development	Set to production for Gin release mode

3. Frontend

cd frontend
npm install
npm run dev

For local dev without Docker, point the Vite proxy at your backend in frontend/vite.config.ts:

proxy: {
  "/api": {
    target: "http://localhost:8080",
    changeOrigin: true,
  },
},

App: http://localhost:5173

---

Project structure

page-pulse/
├── backend/
│   ├── controllers/    # HTTP handlers (auth, URLs, crawls)
│   ├── middleware/     # Auth, rate limiting
│   ├── models/         # GORM models (URL, CrawlResult, Link)
│   ├── routes/         # Route registration
│   ├── services/       # Crawler, validation, SSRF guard, URL cleanup
│   └── utils/          # Logging, API responses, URL enrichment
├── frontend/
│   └── src/
│       ├── components/   # Dashboard, UrlDetails, Header, Login
│       ├── services/     # API client (crawls)
│       └── utils/        # API parsing, errors, URL normalization
└── docker-compose.yml

---

API overview

All JSON responses use a common envelope:

{
  "success": true,
  "message": "Optional human-readable message",
  "data": { },
  "error": "Present when success is false"
}

Send Authorization: Bearer <token> on protected routes (returned from POST /api/auth/login in data.token).

Auth

Method	Path	Description
POST	/api/auth/login	Log in (rate-limited)
POST	/api/auth/logout	Log out
GET	/api/auth/me	Verify token

URLs (authenticated)

Method	Path	Description
POST	/api/urls	Add URL and start crawl
GET	/api/urls	List URLs with summary crawl data
GET	/api/urls/crawl	List URLs (enriched, for dashboard)
GET	/api/urls/:id	Get one URL
GET	/api/urls/:id/crawl	Get URL + latest crawl result
DELETE	/api/urls/:id	Delete URL and related data
POST	/api/urls/:id/start	Start / resume crawl
POST	/api/urls/:id/stop	Cancel crawl, set status queued
POST	/api/urls/batch/start	Batch start
POST	/api/urls/batch/stop	Batch stop
POST	/api/urls/batch/rerun	Clear results and re-crawl
DELETE	/api/urls/batch/delete	Batch delete

Rate limits (per client IP): login — 10/min; crawl triggers (add, start, rerun) — 30/min.

URL input rules

• http:// or https:// is preserved when provided; otherwise https:// is added
• www. is not forced (e.g. example.com → https://example.com)
• Private, loopback, and link-local targets are rejected (SSRF guard)

---

Security notes

Do not use default credentials in production. Change auth and use proper secrets management before exposing this app publicly.

• Sessions are stored in memory (lost on restart)
• SSRF checks run at URL validation and before each outbound request during a crawl
• Rate limiting is in-memory per instance (not shared across replicas)

---

Troubleshooting

Issue	Suggestion
Dashboard empty / API errors	Ensure static routes are registered before /:id in the backend (see backend/routes/routes.go)
Frontend cannot reach API locally	Set Vite proxy target to http://localhost:8080 when not using Docker
Port conflicts	Free ports 3306, 8080, and 5173
DB connection fails	Match DB_* env vars between backend and MySQL; wait for MySQL to be ready
Crawl stuck on running	Use Stop or Re-analyze; stop cancels the background crawl
Broken link false positives	Some servers reject HEAD requests; links may be marked broken incorrectly

---

License

See repository license file if present.