Test Suite Documentation

Version: 1.0.0

Release Date: 2025-10-14

Author: Simon Tin-Yul Kok

Overview

Comprehensive test suite for the KnowBe4 GraphQL MCP Server, providing unit tests, integration tests, and security tests for all core components.

Total Test Cases: 248

Test Pass Rate: 100% (248/248 passing)

Test Coverage: 82%+

Execution Time: <10 seconds

Quick Start

# 1. Install test dependencies
pip install -r setup/requirements-test.txt

# 2. Navigate to tests directory
cd tests

# 3. Run all tests
./run_tests.sh

# Or use pytest directly
pytest . -v

Test Coverage Summary

Components Tested

Component	Test Cases	Coverage	Status
GraphQLTools	50+	~85%	✅ Complete
AuditLogger	30+	~90%	✅ Complete
ConversationLogger	35+	~85%	✅ Complete
QueryLibrary	30+	~80%	✅ Complete
Main (Advanced Features)	58	~85%	✅ Complete
Formatters & Helpers	45+	~80%	✅ Complete

Total: 248 test cases, 82%+ coverage

Security Tests Implemented

✅ Mutation Blocking - All mutation queries blocked ✅ PII Field Detection - Multi-strategy detection (regex, word boundaries, aliases) ✅ Rate Limiting - Sliding window implementation (100 queries/60 seconds) ✅ Query Size Limits - Maximum 100KB, 150 lines ✅ Variable Hashing - No plaintext in audit logs ✅ Parameter Hashing - Privacy-preserving conversation logs ✅ Response Hashing - Secure audit trails

Test Files

Core Test Suites

test_graphql_tools.py (600+ lines, 50+ tests)
- GraphQL validation and security controls
- PII field detection (email, phone, multiple strategies)
- Rate limiting enforcement
- Query execution with async/await
- Schema introspection (types, fields, enums)
- Security event logging
test_audit_logger.py (400+ lines, 30+ tests)
- Query logging with hashing
- Event logging (info, warning, error)
- Security event logging
- Variables hashing (no plaintext exposure)
- JSON Lines format validation
- Immediate log flushing
test_conversation_logger.py (450+ lines, 35+ tests)
- Conversation flow tracking
- Tool call logging with parameter hashing
- Processing step tracking
- Full response storage
- Error handling (standalone and in-conversation)
- Duration calculation
test_query_library.py (350+ lines, 30+ tests)
- Query retrieval by name
- Cache set/get/remove operations
- TTL expiration
- Double TTL verification (security feature)
- Cache statistics
- Query templates and categories
test_main.py (665 lines, 58 tests)
- Advanced intelligence features
- UserPreferences class (22 tests)
- NLP extraction (12 tests)
- Predictive suggestions (10 tests)
- Search helpers (4 tests)
- Data validation (10 tests)

Configuration Files

conftest.py (324 lines) - Pytest fixtures, mock schema, test utilities
pytest.ini (50 lines) - Pytest configuration, coverage settings, test markers
run_tests.sh - Convenient test runner with multiple modes

Running Tests

Test Modes

All Tests with Coverage

cd tests
./run_tests.sh all
# Or: pytest . --cov=../mcp-server --cov-report=html --cov-report=term

Unit Tests Only

cd tests
./run_tests.sh unit
# Or: pytest . -m unit -v

Security Tests

cd tests
./run_tests.sh security
# Or: pytest . -m security -v

Integration Tests

cd tests
./run_tests.sh integration
# Or: pytest . -m integration -v

Fast Tests (Exclude Slow)

cd tests
./run_tests.sh fast
# Or: pytest . -m "not slow" -v

Specific Test File

cd tests
./run_tests.sh specific test_graphql_tools.py
# Or: pytest test_graphql_tools.py -v

Specific Test Class

cd tests
pytest test_graphql_tools.py::TestPIIFieldDetection -v

Specific Test Function

cd tests
pytest test_graphql_tools.py::TestPIIFieldDetection::test_pii_field_blocking_email -v

Coverage Reports

Generate HTML Coverage Report

cd tests
./run_tests.sh coverage

# View report
open htmlcov/index.html

Terminal Coverage Report

cd tests
pytest . --cov=../mcp-server --cov-report=term-missing

Coverage Targets

Overall: 80%+ coverage ✅ Achieved (82%)
Core modules: 90%+ coverage ✅ Achieved
Security functions: 100% coverage ✅ Achieved

Test Quality Features

1. Comprehensive Fixtures

Reusable test data across all tests
Mock schema for consistent testing
Temporary directories for isolated file operations
Mock GQL clients to avoid API calls

2. Async Test Support

Proper async/await testing with pytest-asyncio
Mock async context managers
Async query execution tests

3. Security Focus

All security features thoroughly tested
Edge cases and bypass attempts covered
Security event logging verified

4. Isolation

Each test runs independently
No shared state between tests
Temporary directories cleaned up automatically

5. Fast Execution

All tests run in <10 seconds
No network calls (mocked)
No actual file I/O (temp directories)

Writing New Tests

Test Template

class TestNewFeature:
    """Test description."""

    def test_basic_functionality(self, fixture_name):
        """Test basic case."""
        # Arrange
        input_data = {"key": "value"}

        # Act
        result = function_under_test(input_data)

        # Assert
        assert result["success"] is True

    def test_error_case(self, fixture_name):
        """Test error handling."""
        with pytest.raises(ExpectedException):
            function_under_test(invalid_input)

Async Test Template

@pytest.mark.asyncio
async def test_async_function(self, mock_client):
    """Test async functionality."""
    result = await async_function_under_test()
    assert result is not None

Parametrized Test Template

@pytest.mark.parametrize("input,expected", [
    ("input1", "expected1"),
    ("input2", "expected2"),
    ("input3", "expected3"),
])
def test_multiple_cases(self, input, expected):
    """Test with multiple input cases."""
    result = function_under_test(input)
    assert result == expected

Test Markers

Mark tests with categories:

@pytest.mark.unit
def test_unit_level():
    """Unit test for single function."""
    pass

@pytest.mark.integration
def test_integration():
    """Integration test across components."""
    pass

@pytest.mark.security
def test_security_control():
    """Security-focused test."""
    pass

@pytest.mark.slow
def test_slow_operation():
    """Test that takes >1 second."""
    pass

Integration with Development Workflow

Pre-Commit

# Run tests before committing
cd tests
./run_tests.sh fast
cd ..
git add .
git commit -m "Your commit message"

Continuous Integration

Create .github/workflows/tests.yml:

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: pip install -r setup/requirements-test.txt
      - name: Run tests
        run: cd tests && ./run_tests.sh coverage
      - name: Upload coverage
        uses: codecov/codecov-action@v3

Debugging Failed Tests

Verbose Output

cd tests
pytest test_file.py -vv

Show Local Variables

cd tests
pytest . --showlocals

Stop on First Failure

cd tests
pytest . -x

Run Last Failed Tests

cd tests
pytest . --lf

Print Output

cd tests
pytest . -s

Debug with PDB

cd tests
pytest . --pdb

Best Practices

DO:

✅ Write tests before fixing bugs (TDD) ✅ Use descriptive test names ✅ Test edge cases and error conditions ✅ Mock external dependencies ✅ Keep tests fast and isolated ✅ Aim for high coverage (80%+) ✅ Run tests before committing

DON'T:

❌ Test implementation details ❌ Write tests that depend on each other ❌ Make network calls in unit tests ❌ Commit with failing tests ❌ Skip security tests ❌ Test framework code

Troubleshooting

Import Errors

# Ensure mcp-server is in PYTHONPATH
export PYTHONPATH="${PYTHONPATH}:${PWD}/mcp-server"

Async Errors

# Install pytest-asyncio
pip install pytest-asyncio

Coverage Not Working

# Install coverage plugin
pip install pytest-cov

Tests Hanging

# Add timeout
cd tests
pytest . --timeout=30

Key Testing Achievements

✅ Security Hardening Validated

All H1-H3 fixes verified with tests
M1-M5 fixes validated
Security controls working as designed

✅ Regression Prevention

Tests catch breaking changes immediately
Security controls cannot be accidentally removed
API changes break tests first

✅ Documentation Through Tests

Tests serve as usage examples
Expected behavior clearly documented
Edge cases explicitly covered

✅ Confidence for Refactoring

Safe to refactor with test coverage
Changes validated automatically
Breaking changes caught early

Maintenance

Regular Tasks

Daily: Check test pass rate
Weekly: Run full test suite (cd tests && ./run_tests.sh)
Before commits: Run affected tests (cd tests && ./run_tests.sh fast)
Before releases: Run all tests + coverage (cd tests && ./run_tests.sh coverage)
Monthly: Review and update tests
Quarterly: Refactor test code

Test Health Metrics

Monitor:

Test pass rate (target: 100%) ✅ Current: 100%
Code coverage (target: 80%+) ✅ Current: 82%
Test execution time (target: <2 minutes) ✅ Current: <10 seconds
Flaky tests (target: 0) ✅ Current: 0

Resources

Metrics & Goals

Current Status

✅ 248 test cases implemented
✅ ~3,000 lines of test code
✅ 82% coverage achieved
✅ 100% security tests coverage
✅ <10 seconds execution time

Goals Achieved

✅ Comprehensive unit test coverage
✅ All security controls tested
✅ Fast test execution
✅ Easy to run and maintain
✅ Clear documentation

Conclusion

A production-ready, comprehensive test suite has been implemented for the KnowBe4 MCP Server. The test suite:

Validates all security controls (mutation blocking, PII filtering, rate limiting)
Prevents regressions through comprehensive coverage
Enables confident refactoring with automated validation
Documents expected behavior through clear test cases
Executes quickly (<10 seconds) for rapid feedback

The test infrastructure is now in place to support ongoing development with confidence and quality.