A model can translate a riddle perfectly and still miss the answer: the clue may depend on a local custom, familiar object, or shared cultural association. That gap is what makes this benchmark useful—it tests whether a system can apply cultural knowledge in context, rather than merely produce fluent multilingual text.
Key Value
- Community-authored clues: 167 native speakers wrote original riddles across 62 communities, 51 languages, and 45 countries. The riddles are designed to resist lookup and memorization.
- Evaluation beyond exact matching: The release includes reference answers, 170,520 answers from 28 model configurations, and native-speaker annotations. Scores distinguish incorrect, uncertain, partially correct, and correct answers.
- Visible variation across communities: Some communities remain especially challenging; even frontier models reach at most 22% accuracy in certain communities. This highlights why a single aggregate multilingual score can hide cultural blind spots.
- Useful evaluation artifacts: Separate tables provide riddles, model outputs, human labels, and model configuration details, allowing analysis of error types as well as accuracy.
Who It’s For—and the Trade-off
A good fit for researchers studying multilingual and culturally grounded model evaluation, or for teams checking where a model’s knowledge fails across communities. It is not a training resource: the evaluation-only terms prohibit training, fine-tuning, distillation, and synthetic-data generation from the riddles, answers, or labels. Results also depend on human judgments and should not be read as a complete measure of cultural understanding.
