SynthCorp: A Platform for Creating Benchmarks for Data Agents

Introduction

The rapid progress of large language model (LLM)‑driven data agents has outpaced the development of robust benchmarks. Existing test suites quickly saturate and often fail to isolate specific capabilities such as handling evolving schemas, cross‑asset queries, or causal provenance. SynthCorp addresses this gap by providing a programmable environment that synthesizes realistic corporate data estates—combining relational tables, document stores, and data catalogs—and automatically generates evaluation tasks with verifiable gold solutions.

How SynthCorp Works

1. Scenario‑driven event synthesis – Users specify a business scenario (e.g., supply‑chain disruption) and target complexity dimensions (schema evolution, cross‑domain joins, etc.). SynthCorp creates reusable event types (e.g., new contract ingestion, data‑quality incident) and compiles them into programs built from a small set of simulation primitives.
2. Temporal scheduling – Event instances are placed on a virtual calendar. Routine operations (daily sales uploads) intermix with disruptive events (system outages), ensuring that the resulting data estate evolves in a coherent, time‑ordered fashion.
3. Estate materialization – Executing the calendar populates the relational database, document store, and catalog while logging every operation. The log captures causal provenance, enabling downstream verification.
4. Task generation and gold derivation – From the simulation log, SynthCorp automatically creates analytical tasks (e.g., “find the total revenue for customers whose contracts were renewed after a price‑increase event”). It then derives executable gold programs by replaying the logged operations, eliminating the need for manual annotation.

Benchmark Highlights

• Scale and diversity – The initial release includes seven data estates, each up to 1,000 operational tables, spanning five business domains (finance, logistics, HR, marketing, and R&D). Across these estates, 256 analytical tasks probe a range of capabilities, from simple look‑ups to multi‑event causal reasoning.
• Deterministic integrity checks – Every generated estate passes pipeline validations such as foreign‑key consistency, declared‑write compliance, and recurrence constraints, resulting in a near‑zero error rate.
• Cross‑asset complexity – Tasks deliberately require joins across relational, document, and catalog assets, testing agents’ ability to navigate heterogeneous data sources.
• Model discrimination – Evaluations of three representative agents across four data‑context settings demonstrate clear separation between model tiers, confirming that the benchmarks surface meaningful performance differences.

Practical Takeaways for Researchers and Practitioners

• Customizable difficulty – By adjusting event frequency, schema‑change intensity, or the proportion of disruptive events, users can tailor benchmarks to specific research questions or product roadmaps.
• Provenance‑based verification – The simulation log provides a causal chain that can be audited automatically, ensuring that gold solutions are trustworthy without costly human labeling.
• Open‑ended extensibility – SynthCorp’s modular event library encourages the community to contribute domain‑specific event types, expanding the benchmark ecosystem beyond the initial five domains.
• Enterprise relevance – The generated data estates mirror real‑world corporate environments, making performance insights directly applicable to production data‑agent deployments.

Conclusion

SynthCorp offers a systematic, reproducible way to create rich, evolving data estates and associated benchmark tasks. By coupling realistic event‑driven simulations with automatic gold‑program generation, it fills a critical need for robust evaluation of LLM‑powered data agents. Researchers can leverage the platform to stress‑test new architectures, while practitioners gain a realistic testbed for assessing readiness in enterprise settings. The open‑source nature of SynthCorp invites community contributions, promising a continuously expanding suite of challenging benchmarks that will drive the next generation of data‑aware AI systems.