Most large web corpora are mixed and coarse-grained, which hides domain-specific distributions that matter for domain-aware pretraining, evaluation, and dataset curation. FineFineWeb makes those domain boundaries explicit by automatically labeling and refining web data into 67 interpretable domains and releasing statistics and validation splits that let researchers compare domain contributions at scale.
What Sets It Apart
- Iterative coarse→fine pipeline: uses GPT-4 for root-URL labeling, Qwen-instruct models for large-scale content labeling, FastText for coarse recall, and BERT for fine recall—repeated across three rounds to improve domain precision. This workflow balances automated scale with refined quality.
- Domain-first organization: provides per-domain token counts and sample counts (grand total ~4.4 trillion tokens across domains), validation subsets, and domain-domain similarity/duplication analyses so you can select or weight domains for pretraining or evaluation.
- Analysis-ready outputs: includes domain embeddings and benchmark-alignment analyses (MMD/Wasserstein distances, BPC-Acc correlations), enabling empirical decisions about which domains help particular benchmarks or tasks.
Who It's For & Tradeoffs
- Great fit if you need domain-aware pretraining or want to study how domain composition affects LLM/LM performance (e.g., assembling targeted pretraining mixes or probing domain–benchmark alignment). It’s especially useful for experiments that require per-domain sampling, controlled retrieval, or domain-specific validation sets.
- Look elsewhere if you require fully human-curated labels or legal provenance guarantees for every document; the dataset relies on automated labeling and large-scale deduplication and may still contain noise or provenance gaps that require downstream filtering for high-assurance applications.
How It Was Built
- Deduplication: exact dedupe + MinHash to remove redundancy.
- URL & seed labeling: top 1M root URLs labeled with GPT-4 to produce Domain-of-Interest (DoI) and Domain-of-Non-Interest seeds.
- Coarse recall: domain-specific sampling labeled with Qwen2-7B-Instruct to train binary FastText models and recall coarse DoI data.
- Fine recall: coarse DoI data labeled with Qwen2-72B-Instruct to train BERT classifiers, producing refined DoI subsets.
- Iterations: the coarse/fine loop runs for three rounds with updated seeds and frozen BERT to expand and refine domain coverage.
These choices favor scalable, repeatable domain decomposition and statistical analysis over manual curation; expect strong utility for research and large-scale modeling pipelines but plan extra validation for production-critical or provenance-sensitive uses.
