AIAny
Icon for item

Ultra-FineWeb-L1

Provides an L1 filtered English web corpus from recent Common Crawl snapshots for LLM pretraining, including main-text extraction, language and heuristic filtering, sensitive-field replacement, customized cleaning, and MinHash deduplication; contains 1T+ tokens across ~1.14B documents with structured metadata fields.

Introduction

Why this matters

Data quality at web scale directly affects downstream LLM behavior, but filtering large Common Crawl snapshots is costly and error-prone. This dataset offers a reproducible L1 filtering stage that removes boilerplate, sensitive fields, near-duplicates, and common crawl artifacts while preserving wide coverage of contemporary web text, making it a pragmatic foundation for subsequent selection and refinement stages.

What Sets It Apart
  • Snapshot-organized L1 filtering: each Common Crawl dump is processed independently (trafilatura 2.0 for main-text extraction), which preserves provenance and simplifies incremental updates.
  • Practical cleaning pipeline: combines fastText language ID, FineWeb-derived heuristic filters, sensitive-field replacement, targeted mojibake and invisible-character repairs, and MinHash deduplication to yield cleaner, safer documents for pretraining.
  • Scale and provenance transparency: first release covers multiple 2025 CC-MAIN snapshots with ~1.14 billion documents and 1T+ English tokens, stored as parquet parts and annotated with uid, content, meta, and dataset_index fields.
  • Designed for tiered workflows: serves as the L1 layer in the UltraData L0–L4 framework, enabling deterministic downstream L2 classifier selection and L3 refinement.
Who it's for and tradeoffs

Great fit if you need a reproducible, large-scale cleaned web layer to bootstrap model pretraining or to run ablations on cleaning choices; the per-snapshot layout and detailed metadata make tracing and incremental processing straightforward. Look elsewhere if you need already classifier-selected high-quality subsets (use the companion L2 Ultra-FineWeb release) or if you require cross-snapshot global deduplication—deduplication is performed within each dump by design. Also note legal constraints: users must still comply with original source rights and the dataset license prohibits unauthorized unchanged redistribution.

Information

  • Websitehuggingface.co
  • AuthorsJunshao Guo, Shuaikang Xue, Xiaofei Yang, Chuyue Zhou, Qiang Yin, Qingyang Yuan, Qiang Ma, Yuzhou Zhang, Jie Zhou, Chaojun Xiao …
  • Published date2026/08/14

Categories

More Items

Hugging Face

Evaluates whether web agents can complete tasks on live Korean websites. Its 312 human-written tasks span 104 sites, and success is judged by the final website outcome rather than a single prescribed interaction path.

Hugging Face

Provides paired text and synthetic American English speech in two voices for training lightweight TTS models. The 17,371 clips include phoneme-level annotations, with both voices reading each sentence.

Hugging Face

Evaluates offline action prediction for Korean web agents across 549 tasks on 144 websites, pairing screenshots and action histories with human-recorded next-step targets.