Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing

Introduction

Agentic harness evolution seeks to refine an AI agent’s instructions, tools, and control flow by learning from its own task executions. Intuitively, expanding the pool of evolution tasks should boost performance, yet recent observations reveal a paradox: existing approaches actually degrade when the task set grows. This failure stems from suboptimal evidence selection—different subsets of task outcomes (evidence) can lead to markedly different harness updates.

The Evidence Selection Problem

When an agent accumulates execution results across many tasks, the evolution algorithm must decide which pieces of evidence to use for the next optimization round. If the selection is naïve, low‑quality or redundant evidence can dominate, steering the harness toward poorer configurations. The authors’ preliminary study demonstrated that as the task pool expands from 40 to 200 tasks, baseline methods see a drop in pass rate, highlighting the need for a more principled evidence‑search strategy.

Successive Scale Racing (SSR)

SSR addresses this bottleneck with a two‑stage framework:

1. Diverse Evidence Construction – For each evolution round, SSR builds multiple candidate evidence combinations. These are crafted to balance task quality (favoring high‑performing executions) and coverage (ensuring a broad representation of the task space). By varying these dimensions, SSR generates a spectrum of plausible harness updates.
2. Racing Under a Small Compute Budget – Rather than fully validating every candidate, SSR employs a racing mechanism. Candidates are evaluated incrementally; those that fall behind early are pruned, allowing the remaining promising harnesses to receive more thorough testing. This approach dramatically reduces the number of validation executions while still surfacing near‑optimal solutions.

Empirical Highlights

Across three benchmarks and six configuration variants, SSR consistently achieved state‑of‑the‑art results. Notably, as the task pool grew from 40 to 200 tasks, SSR’s pass rate rose from 64.3 % to 68.3 %, whereas all baselines exhibited declining performance. The racing component alone cut validation runs by roughly 60 %, demonstrating that smarter evidence selection can also translate into tangible compute savings.

Practical Takeaways

• Evidence matters more than quantity: Simply adding tasks does not guarantee better harnesses; careful curation of evidence is essential.
• Racing is an efficient selector: Incremental evaluation lets you discard weak candidates early, preserving budget for the most promising harnesses.
• Scalable to larger task pools: SSR’s performance improves with more tasks, suggesting it can handle future expansions of benchmark suites without the degradation seen in earlier methods.

Conclusion

Successive Scale Racing offers a pragmatic solution to the evidence‑selection dilemma in agentic harness evolution. By generating diverse evidence sets and racing them under a constrained compute budget, SSR not only lifts performance as task pools expand but also trims the validation overhead. Researchers and practitioners looking to scale agentic optimization should consider integrating SSR’s adaptive evidence‑search and racing principles into their pipelines.