AIAny
Icon for item

INFINI-NEWS Corpus

Processed, multilingual news corpus of 1.357B articles extracted from Common Crawl CC‑News with per-article WARC provenance. Includes trafilatura-extracted bodies, language labels (GlotLID & CommonLingua), IPTC topic tags, monthly Parquet shards and a companion FM-index for sub-10ms substring queries; bulk text access is gated for academic research.

Introduction

A single obstacle for large-scale media or NLP research is the combined engineering cost of cleaning, labeling, and making terabytes of web news queryable. This release tackles that by delivering pre-extracted article text and lightweight, per-row metadata across ~1.36 billion CC‑News captures so researchers can filter, count, and sample without rebuilding extraction pipelines.

What Sets It Apart
  • Broad multilingual coverage and scale — ~1.357 billion articles spanning Aug 2016–Apr 2026 and hundreds of languages (GlotLID: 1,172 ISO codes; CommonLingua: 331 classes), so cross-lingual studies and longitudinal analyses can proceed without assembling disparate sources.
  • Per-article provenance and reproducibility — every row preserves WARC filename, capture timestamp, and payload digest, enabling exact-traceability back to the original CC‑News WARCs (useful for audits, copyright requests, and deduplication).
  • Rich, ready-to-use metadata — language tags from two independent classifiers, IPTC top‑17 topic labels (with probabilities), parsed publish dates, and other extraction provenance remove the need for downstream classification passes.
  • Query-first workflow via companion FM-index — an ungated FM-index provides sub-10ms exact substring counts and lets you build selective cohorts before pulling parquet shards, drastically reducing I/O and compute costs for exploratory work.
Who It's For and Trade-offs

Great fit if you are an academic researcher or institution wanting large-scale, provenance-preserved news text for NLP training, media studies, longitudinal event analysis, or building search/evaluation corpora. The dataset is tailored for workflows that benefit from pre-extracted bodies and per-row metadata rather than raw WARC processing.

Look elsewhere if you need unrestricted commercial redistribution of original publisher text (the article bodies are gated and released under a research-only access model) or if you require publisher-curated archives with paywalled content and editorial curation not captured by Common Crawl.

Where It Fits

Use this corpus when you want a reproducible, queryable CC‑News-derived resource that trades raw publisher licensing for a research-friendly access model, and pair it with the public FM-index for rapid exploratory queries before requesting gated parquet shards.

Information

  • Websitehuggingface.co
  • OrganizationsUniversity of Graz, Hugging Face
  • AuthorsRuggero Marino Lazzaroni, Jana Lasser, Kirill Solovev
  • Published date2026/01/29

Categories

More Items

Scales verifiable gameplay data by having a Builder and visual and coding players co-develop web games. Progress-weighted training improves agent performance on both its own benchmark and an external game benchmark.

Hugging Face

Provides executable software-engineering repair tasks for evaluating coding agents. Its 2,955 synthetic instances span 53 Python repositories and include bug-injection patches, reference repairs, and test information.

Hugging Face

Pairs multilingual documents with JSON Schemas and target extractions for training and evaluating structured information extraction. Its held-out document types make it possible to test generalization beyond the tasks seen during training.