When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment

Code and prompts for the paper "When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment".

Structure

• Explore_Overrating/code/overrating-behavior/: pointwise and pairwise judging runners.
• Explore_Overrating/code/evaluation/: evaluation and analysis scripts.
• Explore_Overrating/prompts/: prompt variants used in experiments. Prompts in this folder are the exact text variants used for the runs reported in the paper.

Pointwise judging

Run graded or binary judging on (qid, pid) pairs.

python pointwise_main.py \
  --pred graded \
  --model model-name \
  --output outputs/graded.jsonl \
  --query_path data/queries.tsv \
  --passage_path data/collection.tsv \
  --qrels_path data/qrels.txt

Pairwise judging

Run pairwise judging on (qid, pid1, pid2) pairs. Outputs two files: _a.jsonl and _b.jsonl.

python pairwise_main.py \
  --model model-name \
  --output outputs/pairwise.jsonl \
  --query_path data/queries.tsv \
  --passage_path data/collection.tsv \
  --pairs_path data/pair_qrels.txt

Evaluation

Pointwise relevance evaluation:

python evaluate_relevant.py \
  --mode graded \
  --path outputs/graded.jsonl

Pairwise combine and evaluation:

python pair_combine.py \
  --path-a outputs/pairwise_a.jsonl \
  --path-b outputs/pairwise_b.jsonl \
  --out outputs/pairwise.jsonl

python pair_evaluation.py \
  --path outputs/pairwise.jsonl