Can we trust LLMs to judge whether a research idea is novel?

Introduction

Large language models (LLMs) are increasingly being used as novelty judges—systems that decide whether a research idea adds something genuinely new to the scientific literature. A recent benchmark, Old Ideas, Novel Problems, shows that these judges are far from reliable. Small changes in prompts, reasoning budgets, or retrieval settings can swing accuracy by tens of percentage points, and more sophisticated, purpose‑built evaluators often perform worse than a simple Claude Opus 4.6 baseline.

How the Benchmark Was Built

The authors mined ICLR 2026 peer reviews to create two pools of ideas:

• Human‑Only – 154 high‑novelty ideas from top‑10% accepted papers (reviewers explicitly praised originality) and 145 low‑novelty ideas from bottom‑10% rejected papers (reviewers flagged a lack of novelty).
• Human+Generated – The same 154 human ideas paired with 154 ideas generated by a basic LLM prompt. The generated ideas were assumed to be less novel, providing a conservative test of judges on AI‑produced content.

Both pools were validated by human annotators, ensuring that the novelty signals extracted from reviews were accurate.

Core Findings: Instability Across Models and Prompts

Prompt Sensitivity

Across six LLM judges (GPT‑5.1/5.2/5.4, Claude Sonnet 4.5, Claude Opus 4.5/4.6) the biggest performance swings came from changing the prompt wording. For example, GPT‑5.4’s accuracy on the Human+Generated set dropped from 93.5 % to 40.9 % when a detailed prompt was replaced by a simpler one. Even on the Human‑Only set, Claude Opus 4.6’s accuracy shifted by 21 points with the same prompt change.

Model‑Specific Prompt Effects

No single prompt worked best for all models. An instruction that improved Claude Opus 4.6 could hurt GPT‑5.4, and vice‑versa. This means that a prompt tuned for one judge cannot be safely reused for another without re‑evaluation.

Retrieval and Reasoning Budget

Adding extensive literature retrieval or a high reasoning budget did not guarantee better judgments. A high‑budget GPT‑5.6‑Sol agent performed 11 F1 points worse than the cheap Claude Opus 4.6 baseline, while costing roughly ten times more per decision.

Purpose‑Built Evaluators vs. Simple Prompts

Two dedicated novelty‑evaluation pipelines—Idea Novelty Checker and Sakana AI’s AI Scientist—were both less accurate and far more expensive than a straightforward Claude Opus 4.6 prompt with majority voting. The simple baseline achieved a macro‑F1 of 0.854 at a total cost of $2.59, whereas the purpose‑built systems scored 0.642 and 0.772 at costs of $30.66 and $82.05, respectively.

Implications for Automated Scientific Discovery

The brittleness of novelty judges is especially problematic when they are applied to AI‑generated ideas—the very use case they are meant to support. If a judge’s verdict can flip with a few wording changes, researchers cannot trust the evaluation to guide funding decisions, paper submissions, or downstream experimentation.

Toward More Reliable Novelty Evaluation

The authors propose several directions:

1. Test on Targeted Data – Evaluate judges on the exact type of ideas they will assess (human‑authored vs. AI‑generated).
2. Prompt Sensitivity Audits – Systematically measure how verdicts change across a suite of prompts and models.
3. Better Literature Integration – Develop retrieval methods that reliably surface the most relevant prior work without overwhelming the model.
4. Evolving Benchmarks – Continuously update the benchmark with new peer‑review data and emerging scientific domains.

By providing the benchmark, data pipeline, and code, the study offers a foundation for the community to build more robust novelty judges.

Practical Takeaway

If you are deploying an LLM to screen research ideas, start with a simple, well‑documented prompt (e.g., Claude Opus 4.6 with majority voting) and treat its output as a rough signal rather than a definitive verdict. Pair the model’s judgment with human expert review, and regularly audit prompt sensitivity to avoid hidden biases.

--- This article synthesizes the findings of the Old Ideas, Novel Problems study without adding new quotations or unpublished data.