AI Voice Proctoring System

An AI-powered voice proctoring system that monitors and verifies the identity of individuals during online examinations or assessments. It detects unauthorized assistance and impersonation by answering two questions about a recording: is more than one person speaking, and is the person on camera the one producing the audio?

The system is built on three components. All of them run on a standard CPU, with no GPU and no cloud inference.

Component	What it does	Module
WebRTC VAD	Finds the spans of audio that actually contain speech	main/vad.py
Resemblyzer	Compares voice embeddings to detect a second speaker	main/double_voice.py
MediaPipe	Tracks lip motion to check the voice belongs to the person on camera	main/lip_motion.py

WebRTC VAD segments the recording, Resemblyzer verifies each segment against a reference voice, and MediaPipe provides a visual cross-check. Audio containing speech while the face shows no lip motion suggests the voice is coming from off camera.

Installation

This code uses Python version 3.11.9. Create the environment, then run the install script from inside main/:

conda create -n ai-voice-proctoring python=3.11.9
conda activate ai-voice-proctoring
cd main
bash install.sh    # on Windows: install.bat

or with venv:

python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
cd main
bash install.sh    # on Windows: install.bat

The script is two lines, and the order matters:

pip install resemblyzer==0.1.4 --no-deps
pip install -r requirements.txt

Resemblyzer pins old versions of numpy and librosa that conflict with the rest of the stack, so it goes in first with --no-deps and everything else is resolved around it. Installing it the other way round pulls the conflicting pins in before --no-deps can prevent them.

main/requirements.txt is the single dependency list. The # evaluation only block at the bottom is needed just for grid_search.py and evaluate.ipynb. To run the notebook in an IDE you also need ipykernel in the environment.

Quick start

from vad import detect_speech_segments
from double_voice import detect_double_voice

segments = detect_speech_segments("session.wav")
result = detect_double_voice(segments, audio="session.wav")
# Multiple speakers detected: YES
#   [12.4s - 18.9s]

Each module also runs directly:

python main/vad.py main/assets/a_yes_10_2.wav   # speech segments in a file
python main/vad.py                              # live microphone
python main/lip_motion.py                       # webcam lip motion
python main/audio_utils.py extract in.mp4 out.wav

Repository layout

Path	Purpose
main/double_voice.py	Resemblyzer multi-speaker detection, the core module
main/vad.py	WebRTC VAD speech segmentation, file and live microphone
main/lip_motion.py	MediaPipe lip-motion detection, webcam and video file
main/audio_utils.py	Extract 16 kHz audio from video, record a reference clip
main/example_usage.py	Runnable example with tuned parameters
main/grid_search.py	Parameter sweep against the labelled dataset
main/evaluate.ipynb	Evaluation notebook and result plots
main/assets/	Labelled dataset (_yes_ / _no_) plus evaluation output

Limitations

The system verifies voice identity, not phoneme-level audio-visual sync. It can confirm that lips are moving in time with speech, but not that they are forming the sounds being heard, so someone mouthing along convincingly to another person's voice is outside what it detects.

Detection is also tuned for sustained intrusions. On the labelled dataset it catches every intrusion of 10 seconds or more, but only about 60% of 5-second ones, because a brief interruption does not leave enough consecutive evidence to separate it from ordinary embedding noise. See Where the misses are for the full breakdown.

For usage instructions and API documentation, see the Double Voice Detection Module Documentation.