Most TTS benchmarks report objective metrics or isolated MOS ratings — this dataset instead provides the raw human pairwise comparisons behind a large, complete round‑robin preference benchmark, which matters when you want to train or evaluate models that optimize for listener choice rather than proxy scores.
What Sets It Apart
- End-to-end preference evidence: 315,000 eligible pairwise votes collected in a blinded forced‑choice setup, covering every pairing of 15 models across the same 300 prompts (a deterministically complete round‑robin). This yields 31,500 comparison cells and dense overlap useful for ranking and pairwise learning.
- Audio-first exports: the default
audioconfig stores 4,500 unique 48 kHz, mono, 24‑bit FLAC renders once (not duplicated per comparison), so you get the actual stimuli alongside vote records for reproducible training and audit. - Trust-weighting & frozen evidence: each response carries a frozen audio‑trust estimate captured at completion time; raw eligible votes drive the official benchmark while trust‑weighted aggregates are provided for sensitivity analysis.
- Reproducible ranking: the benchmark publishes deterministic Bradley–Terry fits transformed to Elo ratings (category boards + overall board with equal category weighting), plus prompt‑cluster bootstrap uncertainty and a prompt‑level train/test split to avoid prompt leakage when training reward models.
Who it's for & Trade-offs
Great fit if you want to train or evaluate audio preference/reward models, reproduce a large TTS leaderboard, study human vote aggregation/filtering, or compare TTS systems by operational prompt categories (transactional readouts, empathy, names/spell‑outs, troubleshooting, etc.). Look elsewhere if you need broad language coverage (this set is fixed to English prompts), multi‑speaker natural recordings, or longitudinal provider-version metadata (model folder names are stable identifiers but exact provider revision/parameters are not supplied and must not be inferred).
Practical notes and constraints
- Licensing: votes, prompts, and Datapoint‑authored metadata are CC‑BY‑4.0. Audio files are third‑party model outputs; their reuse is governed by each model provider’s terms and may be restricted. Annotator identifiers are salted hashes — do not attempt re‑identification. Access to the full dataset (audio) is gated and requires an authenticated Hugging Face token per the dataset owner’s access policy.
