Automatic Construction of Strong Simple Baselines

Introduction

When a new deep‑learning method is presented, its value is judged against appropriate baselines. In the physical and life sciences, many reported improvements have later been traced to missing or weak baselines, undermining confidence in the results. A recent study investigates whether large language models (LLMs) can automate the construction of simple, yet competitive, baselines for scientific machine‑learning papers.

Building baselines with language models

The authors assembled 36 supervised classification and regression tasks drawn from peer‑reviewed studies published in Nature, Science, and Cell family journals. Each study introduced a novel deep‑learning approach for a scientific application. Using an LLM, the team prompted the model to generate straightforward baseline methods—typically linear models, decision trees, or simple feature‑engineered predictors—based solely on the task description and any available contextual information.

How strong are the automatically generated baselines?

Performance was measured against two reference points: a trivial predictor (e.g., random guessing or constant output) and the reported deep‑learning model. The automatically constructed baselines recovered a median of 95.5 % of the performance gap between the trivial predictor and the deep model, with a mean recovery of 85.7 %. In 12 of the 36 studies, the simple baseline actually matched or outperformed the deep‑learning model, demonstrating that even minimal, automatically generated methods can be surprisingly effective.

Factors influencing baseline quality

The authors explored several dimensions that affect baseline performance:

• Scientific domain: Baselines tended to be stronger in domains where the underlying data structure is well‑captured by linear or tree‑based models (e.g., certain bio‑informatics tasks) and weaker where deep architectures exploit complex spatial or temporal patterns.
• Publication venue: Studies from higher‑impact journals showed slightly larger gaps, suggesting that more ambitious deep‑learning claims may benefit from stronger baselines.
• Test‑time compute: Providing the LLM with additional compute resources (longer inference time, larger model variants) modestly improved baseline quality, but even modest‑size models produced useful results.
• Contextual information: Access to richer study context—such as detailed methodology sections or supplemental tables—allowed the LLM to tailor baselines more closely to the original task, boosting performance.

Practical implications for peer review

Human reviewers struggle to keep pace with the accelerating volume of machine‑learning research, especially when evaluating methodological rigor. Automated baseline construction offers a scalable, reproducible tool that can flag studies where reported improvements may be marginal or ill‑justified. By integrating LLM‑generated baselines into the review workflow, editors and reviewers can obtain an immediate sanity check without needing deep expertise in every scientific subfield.

Conclusion

The study demonstrates that large language models can reliably generate strong simple baselines across a diverse set of scientific machine‑learning tasks. These baselines capture most of the performance advantage claimed by sophisticated deep‑learning models, and in a notable fraction of cases they surpass them. As the scientific community seeks to maintain rigor amid rapid methodological advances, automated baseline generation emerges as a practical, low‑cost safeguard that complements traditional peer review.