AI text detection for Bouncer

Training and evaluation code for the two AI-text detection models used in Bouncer.

Reliable AI-text detection needs models trained specifically for the task — zero-shot LLM prompting is barely better than random, especially for smaller models. Both models here are trained the same way: fine-tune an LLM backbone with LoRA plus a small classification head over the last-token hidden state, supervised by a continuous edit-distance score bucketed into four edit-intensity classes.

The two models

qwen_3_4b/ — server-side model

This is what runs when AI detection happens in our cloud. QLoRA over Qwen/Qwen3-4B (NF4 4-bit backbone, LoRA on all attention + MLP projections). Achieves 99.8% accuracy distinguishing human from AI-generated text on our test set, and 91.7% on the 3-way task (human / AI-edited / AI-generated).

See qwen_3_4b/README.md for how to reproduce, run inference, or retrain.

gemma_4_e2b/ — on-device model

This is what runs entirely on your iPhone when you enable on-device AI detection in Bouncer. Gemma 4 E2B's mobile QAT checkpoint stays frozen; attention LoRA and a linear-probe head are trained under deployment-exact fake-quant (static int8 activations + int8 KV cache) so the classifier survives conversion to the phone's int8 arithmetic. The result deploys as a LiteRT-LM .litertlm bundle plus a ~10 MB adapter and ~40 KB head. Achieves 99.6% on human vs AI-generated and 88.2% on the 3-way task — measured under the same int8 arithmetic the phone runs.

See gemma_4_e2b/README.md for training, eval, the full .litertlm export pipeline, and on-device integration notes.

Comparing the two

Mode	Server (Qwen)	On-device (Gemma)
Human vs AI-generated	0.998 / 0.998	0.996 / 0.996
Human vs (AI-edited or generated)	0.955 / 0.949	0.933 / 0.925
AI-generated vs (human or edited)	0.962 / 0.958	0.948 / 0.942
Ternary	0.917 / 0.916	0.882 / 0.881

Accuracy / macro F1. The on-device model gives up a little accuracy on the harder tasks in exchange for running fully on-phone.

The dataset

We built our own dataset (86k rows, three roughly equal parts human / AI-edited / AI-generated) using the following pre-2022 human sources: Amazon reviews, Yelp reviews, Reddit, FineWeb-EDU, news articles, and Twitter. AI-edited and AI-generated counterparts were produced with Claude Sonnet 4.6, GPT-5.3 Chat, and Gemini 3 Flash.

Available on HuggingFace as DarrenJiaImbue/ai-detection-demo-dataset.

Setup

Requires Python 3.10+ (numpy/torch's minimum on the pins here) — check python3 --version before creating a venv; on some systems the default python3 still resolves to something older (e.g. 3.8), which fails to resolve these requirements at all rather than installing something broken.

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

The .litertlm export toolchain for the on-device model needs a second, separate environment — see gemma_4_e2b/export/requirements-export.txt.

Troubleshooting: torch / CUDA driver mismatch

requirements.txt and requirements-export.txt both pin an explicit torch==<version>+cu126 build. That cu126 tag was chosen to match the CUDA version this repo's own dev boxes report (nvidia-smi's "CUDA Version" field, driver-capped at 12.8) — a plain unpinned torch install resolves to the newest default wheel instead, which may need a newer CUDA runtime than your driver supports, and torch.cuda.is_available() will silently be False (with a driver-version warning on import) rather than raising an install error.

If that happens on your machine:

1. Run nvidia-smi and note the reported CUDA version.
2. Reinstall torch from the matching index, e.g. for CUDA 12.4: pip install torch==<version> --index-url https://download.pytorch.org/whl/cu124 (browse https://download.pytorch.org/whl/<tag>/torch/ — cu121/cu124/cu126/cu128 etc. — for what's available per torch version).

Hardware

Training (gemma_4_e2b or qwen_3_4b) is documented and tested on 8×24GB GPUs; per-GPU VRAM and GPU count aren't hard requirements of the code (both train.py scripts run under plain torchrun --nproc_per_node N for any N, and qwen_3_4b/train.py also runs single-process), but smaller/fewer GPUs will need a smaller per_device_train_batch_size and/or more gradient_accumulation_steps in config.yaml to fit. Expect ~10GB for the dequantized gemma_4_e2b base checkpoint, plus a few GB per training checkpoint and (for the on-device export path) ~2.2GB for the base .litertlm bundle.

License

Code, model weights, and dataset content are all released under CC BY-NC-SA 4.0 — free to use for non-commercial research, share-alike on derivatives.