A self-driving QA system that crawls websites, logs in automatically, generates test cases, detects bugs, tests API endpoints, and produces Allure reports — powered by Playwright, Python, Groq (cloud) or Ollama (local), and Allure.
# 1. Install dependencies
pip install -r requirements.txt
playwright install chromium
# 2. Get a free Groq API key at console.groq.com
# Set it as an environment variable (never put it in config.env):
# Windows:
set GROQ_API_KEY=gsk_your_key_here
# Mac/Linux:
export GROQ_API_KEY=gsk_your_key_here
# 3. Set your target URL in config.env
TARGET_URLS=https://your-site.com
# 4. Run
python run_smart.py# 1. Install dependencies
pip install -r requirements.txt
playwright install chromium
# 2. Start Ollama
ollama pull llama3.2
ollama serve # in a separate terminal
# 3. Set your target URL in config.env
TARGET_URLS=https://your-site.com
# 4. Run
python run_smart.pyReports open automatically when the run completes.
The framework supports two backends. Switch by setting/unsetting GROQ_API_KEY.
| Groq | Ollama | |
|---|---|---|
| Setup | Free API key at console.groq.com | Install Ollama locally |
| TC generation | ~1–2s per call | ~60–300s per call |
| Navigation | ~0.5–1s per call | ~40–130s per call |
| Run time (2 sites) | ~11 minutes | ~51 minutes |
| Visual detection | ❌ Not supported | ✅ via llava |
| Data privacy | Sends prompts to Groq cloud | Fully local, nothing leaves machine |
| Cost | Free tier: 6000 req/day | Free — runs on your hardware |
| Parallel agents | ✅ Truly parallel (no lock) |
Hybrid mode (recommended): Set GROQ_API_KEY for fast text calls + keep Ollama running for visual detection (llava). The framework automatically routes text to Groq and vision to Ollama.
| Capability | Description |
|---|---|
| 🔐 Auto Login | Detects login forms and authenticates — handles Cloudflare, multi-step login, cookie banners, SSO detection |
| 🕷️ Smart Crawl | Discovers and prioritises pages — checkout, auth, and forms visited first |
| 🧠 AI Navigation | LLM decides what to explore next — no fixed scripts |
| 🧪 TC Generation | Generates 5 specific test cases per page based on actual UI elements — saved to Excel |
| 🐛 Bug Detection | Signal-gated — LLM only fires when console errors, failed requests, or DOM errors exist |
| 🔌 API Testing | Captures all XHR/fetch calls during crawl, tests each endpoint directly |
| 👁️ Visual Detection | Optional llava vision model catches layout breaks and broken images (Ollama only) |
| 🔧 Self-Healing | Actions retry with up to 5 fallback strategies when selectors shift |
| 🔄 Regression Stories | Auto-generates YAML regression stories from discovered TCs |
| 📊 Allure Reports | Clean, flat reports — bugs + screenshots + TCs visible without digging |
| 📋 Run Logs | Full timestamped log saved per run alongside bug reports |
# Standard run
python run_smart.py
# With CLI overrides
python run_smart.py --level 2 # autonomy level (1/2/3)
python run_smart.py --urls https://your-site.com # override URL
python run_smart.py --model llama3.2:latest # override Ollama model
python run_smart.py --pages 5 --steps 4 # more coverage
python run_smart.py --agents 2 # parallel agents
python run_smart.py --check # pre-flight only
python run_smart.py --clear-cache # clear LLM cache then run
# View Allure report
allure serve allure-results| Level | Mode | Features | Use when |
|---|---|---|---|
1 |
Manual | Pre-written stories only — no AI calls | Daily CI regression, no API key |
2 |
Semi-Auto | AI navigation + TC gen + signal-gated bugs | Daily smoke testing (recommended) |
3 |
Full Auto | Everything + visual detection + story gen | Weekly full exploration |
TARGET_URLS=https://www.saucedemo.com # comma-separated, no spaces
AUTONOMY_LEVEL=2 # 1=manual, 2=semi, 3=full
HEADLESS=true
BROWSER=chromium
MAX_STEPS=3
MAX_CRAWL_PAGES=3
MAX_CRAWL_DEPTH=2
PARALLEL_AGENTS=1
# ── Groq (fast cloud — recommended) ──────────────────────────────────────────
# Set GROQ_API_KEY as an environment variable — never put it here
# Get free key: https://console.groq.com
GROQ_MODEL=llama-3.3-70b-versatile
# ── Ollama (local fallback + vision) ─────────────────────────────────────────
# Used automatically if GROQ_API_KEY is not set
# Always used for visual detection (llava) regardless of Groq setting
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=llama3.2:latest
OLLAMA_READ_TIMEOUT=400
# ── API Testing ───────────────────────────────────────────────────────────────
API_TESTING=true # test captured XHR/fetch endpoints
API_TIMEOUT_MS=3000 # response time budget in ms
# ── Login ─────────────────────────────────────────────────────────────────────
LOGIN_EMAIL=your@email.com
LOGIN_PASSWORD=yourpassword
# ── Features ──────────────────────────────────────────────────────────────────
STEALTH_MODE=true # bypass bot detection
SELF_HEALING=true # true=exploratory, false=strict
CACHE_ENABLED=true # cache LLM responses 24h
STORY_ENABLED=false # auto-generate regression storiesSecurity: Never commit
config.envwith real credentials. Add it to.gitignore. SetGROQ_API_KEYandLOGIN_PASSWORDas OS environment variables instead.
ai_tester_project/
├── run_smart.py ← Main entry point (use this)
├── run_agents.py ← pytest orchestrator
├── config.py / config.env ← Settings (no secrets here)
├── conftest.py ← pytest + Allure setup
├── pytest.ini
├── requirements.txt
├── Jenkinsfile ← CI/CD pipeline (Groq + local)
│
├── api/
│ └── api_tester.py ← Captures + tests XHR/fetch endpoints
│
├── core/
│ ├── autonomy.py ← Level 1/2/3 feature flag controller
│ └── cache.py ← LLM response cache
│
├── agents/
│ ├── agent_controller.py ← Multi-page crawl loop
│ ├── ai_agent_worker.py ← Per-page: TC gen + bug detect + actions
│ ├── story_generator.py ← Auto-generates regression stories
│ └── story_runner.py ← Executes YAML stories
│
├── ai/
│ ├── ollama_client.py ← Dual-backend: Groq + Ollama auto-routing
│ ├── bug_detector.py ← Signal-gated bug detection
│ └── test_generator.py ← TC generation
│
├── brain/
│ ├── decision_engine.py ← AI navigation decisions
│ ├── action_executor.py ← Self-healing (5 strategies)
│ └── smart_crawler.py ← URL discovery + scoring
│
├── browser/
│ ├── login_handler.py ← Auto-login (20+ selector strategies)
│ ├── dom_extractor.py ← DOM extraction
│ ├── screenshot.py ← Screenshots
│ └── stealth.py ← 12-patch anti-bot fingerprinting
│
├── reporting/
│ ├── bug_reporter.py
│ ├── bug_report_viewer.py
│ ├── testcase_writer.py ← Thread-safe Excel write (race condition fixed)
│ ├── test_reporter.py
│ └── tc_viewer.py
│
└── tests/
├── test_agent_results.py ← Per-agent: bugs + TCs + summary
├── test_api_results.py ← Per-agent API endpoint results
├── test_bugs.py ← One Allure card per bug
├── test_generated_tcs.py ← TCs grouped by page
└── test_user_stories.py ← Story execution
| Card | What it shows |
|---|---|
| 🤖 Agent Run Results | Per-agent: bugs found, TCs generated, duration, screenshots |
| 🔌 API Test Results | Per-agent: endpoints tested, status codes, response times, security |
| 🐛 Bugs Detected | One card per bug — severity, screenshot, error signals |
| 🧪 AI Generated TCs | All TCs grouped by page with Excel download |
| 🔄 Regression Stories | Story execution results (STORY_ENABLED=true only) |
Note: FAILED tests = bugs found. This is intentional — bugs show RED in Allure.
Runs automatically after each crawl. Checks per endpoint:
- Status codes — 5xx = Critical, unexpected 404 = High
- Response time vs
API_TIMEOUT_MSbudget — slow = Medium - Security headers — missing X-Frame-Options, CSP, HSTS = Low
- Sensitive endpoints (user, account, admin) accessible without auth = High
To disable: API_TESTING=false
bug_reports/<run_id>/
bug_001.json ← Browser bug
bug_002.json ← API bug
api_summary_Agent-1.json ← API test summary
bug_report_viewer.html ← HTML bug list
run_<run_id>.log ← Full run log
generated_test_cases/<run_id>/
test_cases.xlsx ← All TCs
tc_viewer.html ← Filterable TC table
screenshots/<run_id>/ ← Per-step + bug screenshots
stories/auto/ ← Auto-generated story YAML files
allure-results/ ← Raw Allure data
allure-report/ ← Generated HTML report
Level 2 run — 1 URL, 3 pages: ~3–5 minutes
Level 2 run — 2 URLs, parallel: ~8–11 minutes
Level 3 run — 2 URLs + vision: ~15–20 minutes
Level 2 run — 1 URL, 3 pages: ~15–20 minutes
Level 2 run — 2 URLs, parallel: ~40–55 minutes
Level 3 run — 2 URLs + vision: ~60–90 minutes
# Groq — no special config needed, fast by default
PARALLEL_AGENTS=2 # safe with Groq (no lock needed)
CACHE_ENABLED=true # skips repeat pages entirely
# Ollama CPU — reduce scope
PARALLEL_AGENTS=1 # sequential avoids timeout race
MAX_CRAWL_PAGES=2 # especially with llava
MAX_STEPS=2 # especially with llava
HEADLESS=true # saves ~200MB RAM✅ Set GROQ_API_KEY as OS environment variable
✅ Set LOGIN_PASSWORD as OS environment variable
✅ Add config.env to .gitignore
✅ Add .env to .gitignore
❌ Never paste real keys into config.env
❌ Never commit config.env to Git
GitHub will block your push and flag the key if it detects a Groq API key in any committed file. If you accidentally commit a key — revoke it immediately at console.groq.com, then scrub history with git filter-repo.
| Problem | Fix |
|---|---|
| Groq 404 error | Check GROQ_MODEL — run curl -H "Authorization: Bearer $GROQ_API_KEY" https://api.groq.com/openai/v1/models to see available model IDs |
| Groq rate limit (429) | Framework auto-retries with backoff — or reduce PARALLEL_AGENTS |
| Groq key not found | Set as OS env var: set GROQ_API_KEY=gsk_... (Windows) or export GROQ_API_KEY=gsk_... (Mac/Linux) |
| Ollama not responding | ollama serve then curl http://localhost:11434/api/tags |
| Ollama model not found | Set OLLAMA_MODEL=llama3.2:latest (exact output of ollama list) |
| Visual detection disabled | Install llava: ollama pull llava — requires Ollama running |
| Excel TC file corrupted | Race condition — fixed in testcase_writer.py with _excel_lock |
| Config prints twice | Remove last line of config.py: print("\n" + CFG.summary() + "\n") |
| Site blocked / Access Denied | Pre-flight check fails but Playwright stealth browser will still reach it |
| SSL warnings in output | Already suppressed — add PYTHONWARNINGS=ignore to env if still showing |
| Allure CLI not found | scoop install allure or allure serve allure-results manually |
| Stories all failing | Stories start from base URL (pre-login) — expected, not a framework bug |
| GitHub blocked push | Groq key in config.env — revoke key, remove from file, add config.env to .gitignore |
See JENKINS_SETUP.md for full CI/CD setup guide.
Jenkinsfile supports both Groq and Ollama backends. Groq API key stored as a Jenkins credential (never in logs). All parameters configurable per-build.
Is: A QA accelerator — finds bugs faster than manual exploration, generates first-draft TCs, gives a starting point for regression coverage.
Isn't: A production CI test suite. The LLM is non-deterministic. Auto-generated stories need human review. Use Level 2 daily, Level 3 for new site exploration only.
Stack: Playwright + Python + Groq / Ollama + Allure. Text via Groq (cloud, fast) or Ollama (local). Vision always via Ollama/llava.