theaicatalog
Hugging Face ยท Research

Ai2 study finds LLM judges of research novelty are unreliable

Ai2 released a paper, benchmark, and data for testing whether large language models can judge if a research idea is novel. The study tested six LLM judges, including GPT-5.1, GPT-5.4, Claude Sonnet 4.5, and Claude Opus 4.6. Ai2 reports that small prompt changes can shift results sharply, and for GPT-5.4 accuracy on one task dropped from 93.5% to 40.9% after a simpler prompt. A simple prompted Claude Opus 4.6 baseline also beat two purpose-built novelty evaluation systems on accuracy and cost.