A single obstacle for large-scale media or NLP research is the combined engineering cost of cleaning, labeling, and making terabytes of web news queryable. This release tackles that by delivering pre-extracted article text and lightweight, per-row metadata across ~1.36 billion CC‑News captures so researchers can filter, count, and sample without rebuilding extraction pipelines.
What Sets It Apart
- Broad multilingual coverage and scale — ~1.357 billion articles spanning Aug 2016–Apr 2026 and hundreds of languages (GlotLID: 1,172 ISO codes; CommonLingua: 331 classes), so cross-lingual studies and longitudinal analyses can proceed without assembling disparate sources.
- Per-article provenance and reproducibility — every row preserves WARC filename, capture timestamp, and payload digest, enabling exact-traceability back to the original CC‑News WARCs (useful for audits, copyright requests, and deduplication).
- Rich, ready-to-use metadata — language tags from two independent classifiers, IPTC top‑17 topic labels (with probabilities), parsed publish dates, and other extraction provenance remove the need for downstream classification passes.
- Query-first workflow via companion FM-index — an ungated FM-index provides sub-10ms exact substring counts and lets you build selective cohorts before pulling parquet shards, drastically reducing I/O and compute costs for exploratory work.
Who It's For and Trade-offs
Great fit if you are an academic researcher or institution wanting large-scale, provenance-preserved news text for NLP training, media studies, longitudinal event analysis, or building search/evaluation corpora. The dataset is tailored for workflows that benefit from pre-extracted bodies and per-row metadata rather than raw WARC processing.
Look elsewhere if you need unrestricted commercial redistribution of original publisher text (the article bodies are gated and released under a research-only access model) or if you require publisher-curated archives with paywalled content and editorial curation not captured by Common Crawl.
Where It Fits
Use this corpus when you want a reproducible, queryable CC‑News-derived resource that trades raw publisher licensing for a research-friendly access model, and pair it with the public FM-index for rapid exploratory queries before requesting gated parquet shards.
