More Rules, Longer Lists? Benchmarking Self-Evolving Agents on a Hidden-Rule Curriculum

Introduction

The paper More Rules, Longer Lists? Benchmarking Self‑Evolving Agents on a Hidden‑Rule Curriculum presents a new evaluation suite, CLR‑Bench, designed to probe whether agents that continuously rewrite their internal state can discover rules that were never explicitly taught. By stacking hidden rules across multiple levels, the benchmark tests an agent’s ability to infer latent groupings—such as the members of an airline alliance—using only sparse feedback.

The CLR‑Bench Curriculum

CLR‑Bench constructs chains of tasks, each adding one hidden rule on top of the previous ones. A rule operates on an opaque label that corresponds to a real‑world named grouping (e.g., founding members of a consortium). For each level the agent receives:

1. One example value (a known member of the group).
2. Practice feedback that reveals an additional member and a non‑member.
3. Test queries that ask for unseen members and similar non‑members.

The benchmark includes an automated pipeline that can generate chains from any new data domain, and the authors release 20 chains covering 10 domains (e.g., airlines, sports leagues, scientific collaborations). This systematic construction ensures that success requires genuine rule discovery rather than memorizing a static list.

Evaluation of Self‑Evolving Methods

Six recent self‑evolution approaches were evaluated on CLR‑Bench, ranging from simple exception‑list updates to more sophisticated dynamic cheatsheets. The authors also introduced a certifying model that knows the true group members but does not guess without feedback, serving as a sanity check.

Key findings:

• Modest gains over a frozen baseline: the best method improved accuracy by at most 5 points, excluding only 6–12 % of unseen members, far below the 20 % exclusion achieved by merely retaining raw practice history.
• Fixed lesson boost: providing a brief instruction about the "named‑grouping" convention lifted all methods by 14.5–16.5 points, highlighting the importance of explicit curriculum framing.
• State composition: agents tended to store the revealed member (75–98 % of cases) while rarely retaining the class name itself, and they correctly added missing members when practice exposed them.
• Stronger models: using a more capable language model (referred to as GPT‑6‑Sol) allowed two methods—ACE and Dynamic Cheatsheet—to reach 78 % accuracy, solving 59–60 % of single‑cell queries, compared with 53 % and 8 % for the certifying model.

Overall, CLR‑Bench demonstrates that current self‑evolving agents learn only shallow patterns unless given clear guidance or a more powerful underlying model.

Practical Takeaways

1. Curriculum design matters – Explicitly stating the nature of hidden rules can dramatically improve learning outcomes.
2. Model capacity limits – Even sophisticated self‑evolution mechanisms hit a ceiling when the base model lacks sufficient reasoning ability.
3. Benchmark utility – CLR‑Bench offers a reproducible, domain‑agnostic framework for measuring rule‑discovery capabilities, useful for both academic research and industry product testing.
4. Future directions – Enhancing agents with meta‑learning or prompting strategies that surface latent groupings could close the performance gap observed in the study.

Conclusion

CLR‑Bench provides a rigorous, scalable way to assess whether self‑evolving agents truly internalize hidden rules rather than merely accumulating exceptions. The results suggest that, without explicit curriculum cues or a more powerful language model, agents struggle to achieve robust rule inference. Researchers and practitioners should consider incorporating clear instructional prompts and leveraging stronger base models when building systems that must learn from minimal feedback.