Why this matters
Traditional Chinese evaluation resources are scarce and often biased toward simplified-character datasets. That gap makes it hard to measure and improve LLM performance for Taiwan-specific language, terminology, and curricula. TMMLU+ provides a larger, more balanced, and region-aware multiple-choice benchmark so researchers can reliably assess and compare models on Traditional Chinese understanding.
What Sets It Apart
- Broad coverage: 66 subjects spanning primary, secondary, undergraduate and professional domains (STEM, social sciences, humanities, and other vocational topics), plus Taiwan-specific items (e.g., Taiwanese Hokkien, geography, culture). This diversity stresses models across factual knowledge, domain terminology, and local context.
- Scale and quality control: v1.1 contains ~22,203 verified multi-choice questions. The release corrected or removed thousands of problematic items via automated scans and human expert review, improving answer keys and removing ambiguous entries.
- Evaluation-ready: pre-split dev/validation/test sets, CSV/Parquet formats and direct integration with the Hugging Face datasets library make benchmarking and reproducing results straightforward. A public leaderboard reports per-category and aggregate accuracies for many commercial and open models.
- Region- and script-aware: focuses on Traditional Chinese wording and Taiwan terminology differences (not just character conversion), so it better reflects real-world usage in Taiwan than Simplified-Chinese-trained benchmarks.
Who It's For and Trade-offs
Great fit if you need a standardized, region-accurate benchmark to evaluate or compare LLMs' Traditional Chinese multiple-choice performance, to identify domain gaps (e.g., law, medical, vocational exams), or to build finetuning/evaluation pipelines that respect Taiwan-specific terminology.
Look elsewhere if your primary goal is open-ended generation, instruction-following dialogue, multimodal evaluation, or large-scale retrieval-augmented QA: TMMLU+ is multiple-choice centric and does not measure free-form generation quality or multimodal reasoning. Also, because it emphasizes Taiwan context, its topical distribution may not represent other Traditional Chinese user populations (e.g., Hong Kong) without additional adaptation.
Practical notes
- Licensing: MIT — permissive for research and commercial use as long as copyright notice is kept.
- Format & tooling: available via Hugging Face datasets with per-subject configs; usable as CSV/Parquet and directly loadable with load_dataset('ikala/tmmluplus', '
<subject>'). - Empirical findings (from the release): Traditional Chinese variants of many models still lag simplified-Chinese-trained counterparts; top closed-source models outperform most open models but average model performance remains below human levels on aggregate scores.
