OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

Introduction

The rapid progress of large language models (LLMs) has sparked interest in their capacity to tackle genuine research challenges. Yet the community lacks a unified, well‑structured set of problems that span multiple scientific domains, as well as a reliable way to evaluate LLMs on tasks that truly require research‑level reasoning. OpenFrontiers fills this gap by providing two linked resources: a massive corpus of open research problems and a formally verified benchmark of recently solved problems, primarily from mathematics. Together they enable systematic study of LLM‑driven scientific rediscovery and offer a baseline for future advances.

A Structured Corpus of Open Problems

OpenFrontiers compiles 12,151 research problems drawn from a variety of fields—mathematics, physics, computer science, and beyond. Each entry records:

• Problem statement: a concise description of the open question.
• Context and background: relevant literature, definitions, and prior attempts.
• Progress tracking: notes on partial results, community interest, and any ongoing efforts.
• Metadata: discipline tags, difficulty estimates, and citation links.

The corpus is designed for easy querying and extension, encouraging researchers to contribute new problems or update existing ones as the scientific landscape evolves.

The Rediscovery Benchmark

From the larger corpus, the authors extracted a benchmark of 200 problems that were resolved in 2026. This subset includes:

• 110 solved problems with publicly available proofs.
• 90 still‑open problems that serve as a test set for future LLM attempts.

A further filtered group, HQ‑50, contains the highest‑quality entries based on a multi‑factor scoring system (clarity, formalization, and verification depth). Each solved problem is encoded in Lean 4, a proof‑assistant language, and is accompanied by automated test probes and two independent human reviews to ensure correctness.

Evaluation Protocol

OpenFrontiers defines three evaluation conditions for LLM agents:

1. Closed‑book – the model must generate a proof without external resources.
2. Frozen‑retrieval – the model can retrieve static documents from a pre‑indexed snapshot of the web.
3. Live‑web – the model may query the current internet, simulating real‑world research assistance.

All generated proofs are run in an isolated sandbox where two independent checkers verify correctness. The protocol also imposes a standardized computational budget to keep comparisons fair.

Findings on Current LLM Capabilities

When seven LLM agents were tested on the HQ‑50 subset:

• Closed‑book success was rare, with at most 7 out of 40 resolved problems correctly reproduced.
• Live‑web performance was higher, but most successes stemmed from retrieving already published solutions rather than novel rediscovery.

These results suggest that, while LLMs excel at information retrieval, genuine reasoning and proof construction at the research level remain challenging.

Implications and Future Directions

OpenFrontiers offers a concrete foundation for several research avenues:

• Benchmarking progress: As LLMs improve, the same benchmark can track incremental gains in true scientific reasoning.
• Tool integration: Researchers can experiment with hybrid systems that combine LLMs, formal proof assistants, and domain‑specific databases.
• Open problem discovery: The corpus itself can be mined to identify promising open questions, prioritize funding, or foster collaborative efforts.
• Community-driven expansion: By inviting contributions, the dataset can stay current with emerging fields and interdisciplinary challenges.

Practical Takeaway

For AI practitioners interested in evaluating or improving research‑level LLMs, OpenFrontiers provides both a rich problem catalog and a rigorously vetted benchmark. Leveraging the Lean 4 formalizations and the defined evaluation protocol enables reproducible experiments that distinguish between mere retrieval and authentic scientific reasoning. As the benchmark evolves, it will serve as a yardstick for the next generation of AI systems aiming to assist, and perhaps eventually lead, in solving humanity’s most demanding open problems.