decontam

Detect benchmark/eval data contamination against your training corpus — and emit a cleaned dataset. Zero dependencies, pure n-gram overlap, CI-gate friendly.

When an evaluation benchmark leaks into training data, your model's scores are inflated and meaningless. decontam finds those leaks the standard way the big labs do — exact word n-gram overlap (the GPT-3 / Llama-style 8–13-gram method) — then flags or removes the contaminated examples.

• 🔍 Finds leaks — overlapping n-grams between an eval set and a training corpus.
• 🧹 Cleans datasets — writes a new eval/training file with leaked rows removed.
• 🚦 CI gate — exits non-zero when contamination is found, so leaks fail the build.
• 📦 Zero dependencies — pure ESM JavaScript, runs on Node 18+.
• 🧩 Library + CLI — use it in a script or from the terminal.
• 📄 Reads .txt, .jsonl, .json — including chat-style messages records.

---

Install

# run without installing
npx decontam --help

# or clone and link locally
git clone https://github.com/Ayubjon/decontam.git
cd decontam
npm link        # exposes the `decontam` command

No build step, no API keys, nothing to configure.

Usage

decontam --train <corpus> --eval <benchmark> [options]

Option	Default	Description
--train <file>	—	Training corpus (.txt / .jsonl / .json)
--eval <file>	—	Eval/benchmark set to check
--n <int>	13	n-gram size (8–13 is typical)
--threshold <0..1>	0.5	flag when the overlap score ≥ threshold
--min-run <int>	—	also flag when the longest leaked token run ≥ this
--field <name>	auto	record field holding the text (eval set)
--train-field <name>	auto	record field holding the text (training corpus)
--out <file>	—	write the cleaned eval set (flagged rows removed)
--report <file>	—	write a JSON report of the run
--json	—	print the JSON report to stdout
--no-fail	—	exit 0 even when contamination is found

Exit codes: 0 clean · 1 contamination detected · 2 usage/IO error.

Quick example

The repo ships with runnable fixtures in examples/:

decontam --train examples/train.jsonl --eval examples/eval.jsonl \
         --n 8 --out examples/eval.clean.jsonl

decontam — n=8, threshold=50.0%, index 52 n-grams
Scanned 6 examples → 3 clean, 3 flagged (50.0% contaminated)

Flagged examples:
  #0  score=100.0%  run=14  (7/7 n-grams)
      "Water boils at 100 degrees Celsius at standard atmospheric pressure measured at…"
  #2  score=100.0%  run=21  (14/14 n-grams)
      "The Pythagorean theorem states that the square of the hypotenuse equals the sum…"
  #4  score=100.0%  run=15  (8/8 n-grams)
      "In 1492, Christopher Columbus crossed the Atlantic Ocean and reached the islands…"

examples/eval.clean.jsonl now contains only the 3 uncontaminated questions.

Use as a library

import { decontaminate } from 'decontam';

const train = ['The capital of France is Paris on the river Seine.'];
const evalSet = [
  { id: 1, text: 'The capital of France is Paris on the river Seine.' }, // leak
  { id: 2, text: 'What is the boiling point of water at sea level?' },   // clean
];

const { clean, flagged, stats } = decontaminate(evalSet, train, {
  n: 8,
  threshold: 0.5,
  text: (item) => item.text,
});

console.log(stats);          // { total: 2, clean: 1, flagged: 1, rate: 0.5, ... }
console.log(flagged[0].score); // 1  → exact overlap

Lower-level building blocks are exported too: NgramIndex, scoreOverlap, ngrams, hashGram, normalize, tokenize, parseRecords, extractText.

How it works

text ──▶ normalize (lowercase, strip punctuation)
     ──▶ word n-grams (sliding window of n tokens)
     ──▶ FNV-1a hash each n-gram  ──▶ Set index over the training corpus
     ──▶ for each eval example: score = (matched n-grams) / (total n-grams)
     ──▶ flag if score ≥ threshold  OR  longest leaked run ≥ --min-run

Storing only 32-bit hashes keeps the index compact, so large corpora fit in memory. The longest-run signal catches long verbatim spans even when a single short n-gram score would look diluted.

Note. n-gram overlap catches verbatim and near-verbatim leaks — the most common and most damaging kind. It does not catch paraphrased or translated contamination; treat a clean decontam run as necessary, not sufficient.

Why n-grams?

It's the method published alongside GPT-3, the Llama models, and most open datasets precisely because it's transparent, deterministic, and cheap. No embeddings, no model calls, no cost — just exact lexical overlap you can audit.

Development

npm test    # node --test, zero dependencies

License

MIT © 2026 Ayubjon

Support

This project is free and open source. If it saved you from a contaminated benchmark, an optional tip is always welcome (never required):

• USDT — Ethereum (ERC-20): 0xad39bdf2df0b8dd6991150fcea0a156150ed19b8
• Verify on Etherscan

Please send only on the Ethereum (ERC-20) network. Thank you! 🙏