Most work on LLM training focuses on models and architectures; DataPrep-Bench asks the opposite practical question: how well can LLMs (and agents built around them) produce the actual supervised data that powers fine-tuning? Testing data-preparation as a measurable engineering task matters because dataset quality directly determines downstream model utility, yet lacked a unified, comparable benchmark.
Key Findings
- Two complementary capabilities are benchmarked: data construction (transforming raw sources into labeled training examples) and data-quality evaluation (predicting a candidate dataset's downstream utility). This makes dataset creation itself a first-class evaluation target.
- The benchmark runs across six domains with a downstream-grounded protocol: outputs are evaluated by fine-tuning base models (jointly with Dolly-15k) and measuring downstream performance, so metric gains reflect real training value, not just surface text quality.
- A released agent, Data-Construction-Skill, improves the Dolly-only baseline by nearly 20 absolute points on Llama-3.1-8B in a Finance task and is competitive with strong agent- and DataFlow-based methods in knowledge-dense domains—showing skill-guided agents can materially raise training utility.
- For quality evaluation, the Distributional Alignment Score (DAS) — an MMD-based distance to a domain proxy — achieves the strongest cross-model correlation in four of six domains and is the only metric that clears r>0.70 simultaneously in Math, Science, and Medical, outperforming existing heuristic, quality- and diversity-based scorers.
Who it's for and tradeoffs
Great fit if you build or evaluate LLM-driven data pipelines (dataset curators, research labs, companies automating instruction/data generation): DataPrep-Bench gives a reproducible, downstream-grounded way to compare data-construction agents and dataset scorers. Use it to validate whether an LLM-generated dataset will actually improve a downstream fine-tuned model.
Look elsewhere if you only need surface-level text quality checks (fluency/grammar) or synthetic-data probes without downstream fine-tuning: the benchmark emphasizes training utility and requires downstream evaluation compute. Its strengths depend on the chosen base models and domain proxies, so absolute scores can shift with different base-model choices.
