Speech-to-speech training often needs more than a transcript: models must learn who spoke, when turns overlap, and how separate voices relate to the mixture. This dataset makes that structure available in token form, but the trade-off is important: it is not a self-contained audio archive, and its tokens are tied to the Mimi representation.
What the Data Enables
- The 3,630 spans total 408.25 hours of English speech selected for little or no music, giving pre-training a long-form speech source rather than isolated utterances.
- In 856 two-speaker spans, each voice has its own token track alongside the mixed track. This directly supports learning speaker-specific output and overlapping conversation.
- Most spans have speaker-attributed transcripts with word timings; some also include emotion labels and vocal-burst annotations. These labels are machine-generated, so they are useful for scale but should not be treated as human-verified ground truth.
- The audio is represented as eight Mimi codebooks at 12.5 Hz, while raw recordings are omitted. Source coordinates can help locate audio in YODAS3, but changed source files may no longer match the recorded hashes.
Who Should Use It—and the Trade-Off
A good fit for researchers pre-training dual-track speech-to-speech systems or studying turn-taking and speaker separation from tokenized data. Look elsewhere if you need raw waveforms, multilingual coverage, codec-independent audio, or carefully human-verified labels. The dataset follows the YODAS3 CC BY 3.0 license.