AI Dataset2024
Hisami Suzuki, Satoru Katsumata +4LLM-jp (Center for Large Language Model Research and Development, National Institute of Informatics), National Institute of Informatics
Provides manually curated Japanese instruction pairs (questions and safe reference answers) for improving LLM output safety, covering broad harm categories and regionally sensitive cases. Includes English meta-tags and standard splits for benchmarking and fine-tuning.
AI Dataset2024
A 12K-question multiple-choice benchmark that raises difficulty by expanding choices to ten and adding reasoning-focused problems. Designed for robust LLM evaluation with cross-domain coverage, expert curation, and emphasis on chain-of-thought performance.
AI Dataset2024
Provides a 1 trillion-token multimodal interleaved dataset (HTML subset updated as data_v1_1 with 742B HTML tokens) and 3.4B images drawn from HTML/PDF/ArXiv sources for multimodal pretraining; released under CC-BY-4.0 with safety and deduplication guidance.
AI Dataset2024
Structured dataset for training and evaluating LLM agentic behavior: function-calling conversations, JSON-mode structured outputs, and extraction samples for teaching models to generate tool calls and strict structured responses. Includes single-turn and multi-turn scenarios across several configs.
AI Dataset2024
A monthly-updated, hive-partitioned parquet corpus of billions of standard rated chess games from Lichess for data analysis and ML — includes move lists with clock info, player Glicko2 ratings, openings, timestamps, and some Stockfish evals under CC0.
AI Dataset2025
Provides 1.7M+ synthetic and real infographic charts paired with their tabular data for training and evaluating multimodal models on infographic understanding, chart-to-table extraction, chart code generation, and example-based chart synthesis.
AI Dataset2025
Provides MS MARCO queries, passages and answers translated into 14 Indic languages while keeping the original English content and per-example translation metadata. Includes train/validation splits, passage selection flags, and translation model parameters for multilingual IR, QA and RAG research.
AI Dataset2025
Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili +1University of Tehran, Institute for Research in Fundamental Sciences (IPM)
Provides a large-scale, multi-speaker Persian speech–text corpus constructed from audiobooks for TTS, ASR, and speaker research. Includes automated alignment and quality scoring, TTS-ready subsets (thousands of hours/1M+ segments) and metadata for speaker IDs and genders — suitable for multi-speaker synthesis and voice cloning research.
AI Dataset2025
Provides a reproducible, deduplicated corpus of text extracted from PDFs for LLM pretraining—about 3 trillion tokens from ~475 million documents in 1733 language-script pairs. Includes OCR and text extraction pipelines, per-page language IDs, MinHash deduplication, and is released under ODC‑By 1.0.
AI Dataset2025
Open synthetic corpus for training small reasoning-focused language models — ~79.65M generated samples (≈75B tokens with Pleias tokenizer) amplified from ~58.7k Wikipedia/Wikibooks seeds; includes explicit synthetic reasoning traces, multilingual coverage, and parquet splits.
AI Dataset2026
Provides 6,000 runnable, operator-level PyTorch tasks for training and evaluating CUDA kernel generation models; each sample includes executable code, operator descriptors, and provenance tags, with execution-driven filtering to ensure reproducibility and contamination control.
AI Dataset2026
Provides a sanitized, labeled SOC capture and merged provenance graph for intrusion-detection research, including 2,011,674 live signals, 51,371 incident graphs, MITRE ATT&CK mappings, and deterministic attack reports.