AI Dataset2026
Provides 1.3 billion platform-specific video URLs extracted from CommonCrawl along with crawl metadata (no media included), serving as the source corpus for the LAION-BVD multimodal video dataset; distributed on Hugging Face in Parquet format.
AI Dataset2026
Provides 55 million scene-level video clips (each with captions, language labels, and timestamps) extracted from an 80M-video, 10-million-hour raw pool to support multimodal pre-training across video, audio, and frames. Access is gated for academic/non-commercial research.
AI Dataset2026
Provides an L1 filtered English web corpus from recent Common Crawl snapshots for LLM pretraining, including main-text extraction, language and heuristic filtering, sensitive-field replacement, customized cleaning, and MinHash deduplication; contains 1T+ tokens across ~1.14B documents with structured metadata fields.
AI Dataset2026
Provides Parquet-backed pretraining subsets of web and synthetic QA text for large-language-model training, including web-high-nltk-qa, web-high-medium, and txt360-qa. Offers streaming access, provenance metadata, and CC BY 4.0 licensing; intended for LM pretraining and research.
AI Dataset2026
A 10‑billion‑document retrieval benchmark with per‑document 768‑dim unit‑norm dense embeddings and mGTE sparse embeddings, FineWeb text/metadata, and exact top‑1000 MS MARCO ground truth for ~120k queries. Built for large‑scale evaluation of dense/sparse/hybrid retrieval, filtered search, indexing, ANNS algorithms, and embedding compression.