Why this matters
Data quality at web scale directly affects downstream LLM behavior, but filtering large Common Crawl snapshots is costly and error-prone. This dataset offers a reproducible L1 filtering stage that removes boilerplate, sensitive fields, near-duplicates, and common crawl artifacts while preserving wide coverage of contemporary web text, making it a pragmatic foundation for subsequent selection and refinement stages.
What Sets It Apart
- Snapshot-organized L1 filtering: each Common Crawl dump is processed independently (trafilatura 2.0 for main-text extraction), which preserves provenance and simplifies incremental updates.
- Practical cleaning pipeline: combines fastText language ID, FineWeb-derived heuristic filters, sensitive-field replacement, targeted mojibake and invisible-character repairs, and MinHash deduplication to yield cleaner, safer documents for pretraining.
- Scale and provenance transparency: first release covers multiple 2025 CC-MAIN snapshots with ~1.14 billion documents and 1T+ English tokens, stored as parquet parts and annotated with uid, content, meta, and dataset_index fields.
- Designed for tiered workflows: serves as the L1 layer in the UltraData L0–L4 framework, enabling deterministic downstream L2 classifier selection and L3 refinement.
Who it's for and tradeoffs
Great fit if you need a reproducible, large-scale cleaned web layer to bootstrap model pretraining or to run ablations on cleaning choices; the per-snapshot layout and detailed metadata make tracing and incremental processing straightforward. Look elsewhere if you need already classifier-selected high-quality subsets (use the companion L2 Ultra-FineWeb release) or if you require cross-snapshot global deduplication—deduplication is performed within each dump by design. Also note legal constraints: users must still comply with original source rights and the dataset license prohibits unauthorized unchanged redistribution.
