Indic DiarBench

Indic DiarBench is intended to fill a practical evaluation gap: multilingual, speaker‑attributed ASR benchmarks that reflect conversational Indian speech are rare, and existing datasets either lack speaker labels, multilingual breadth, or realistic overlap. The core insight is that realistic multi‑speaker evaluation requires both time‑aligned speaker turns and transcription variants that handle code‑mixing — this dataset provides both, with annotation and quality controls designed for speaker‑attributed ASR research.

What Sets It Apart

• Multilingual + speaker attribution: covers all 22 scheduled Indian languages with speaker‑attributed, time‑aligned segments rather than only sentence‑level transcripts, enabling joint diarization+ASR experiments. This is suitable for evaluating models that must both segment speakers and produce text per speaker.
• Realistic acoustic variety and overlap: combines near‑field (per‑speaker mics), far‑field (distant mics with reverberation/noise) and in‑the‑wild (YouTube) samples so models can be stress‑tested across capture conditions; average overlap ratio is nontrivial, so overlap handling matters in evaluation.
• Human-in-the-loop, code‑mix aware annotations: initial ASR bootstraps are fully human‑validated, and transcripts include native‑script and normalized formats to support different downstream pipelines and code‑mixed evaluation.
• Practical dataset design choices: clips derive from 590 source recordings (not 1,164 independent recordings) — several recordings contribute multiple clips and one contributes 21 — so grouping by recording_id is essential when splitting or aggregating metrics to avoid leakage.

Who It's For and Tradeoffs

Great fit if you need a reproducible benchmark to evaluate combined diarization+ASR systems on Indian languages, study code‑mixing robustness, or compare commercial APIs and research models on speaker‑attributed transcription. It is also useful for development of multilingual ASR models with limited per‑language training resources. Look elsewhere if you require large per‑language corpora for full ASR training (many languages have only ~1–4 hours each), or if you need fully independent utterance‑level samples (clips from the same recording share speakers and acoustic conditions). Note also that far‑field and in‑the‑wild conditions only cover subsets of languages (far‑field: top 8 languages; in‑the‑wild: 10 most widely spoken languages).

Practical notes for researchers

• Use sample_id as the unique key for per‑sample reporting, but group splits and cross‑validation by recording_id to avoid correlated samples dominating results.
• The dataset ships with evaluation baselines (DER, cpWER, WDER) and duration‑weighted aggregate metrics; those are convenient reference points when comparing new models or APIs.
• License: CC BY 4.0 — permissible for research and redistribution with attribution.