This dataset exists because data composition strongly shapes embedding quality: reproducing the mGTE/LateOn recipe enables controlled comparisons and incremental improvements when pre-training retrieval/embedding models. Instead of a single monolithic dump, the collection exposes many source-specific configurations and per-row metadata so researchers can test how each source and cleaning decision affects downstream retrieval performance.
What Sets It Apart
- Source-granular configs: Each contributing corpus (Wikipedia variants, MS MARCO, arXiv title/abstracts, Reddit, FineWeb subsets, and many more) is provided as an independent configuration so you can include/exclude or weigh sources precisely — this enables ablations and reproducible experiments. So what: you can isolate the effect of one source (e.g., web crawl vs. curated QA) on embedding retrieval metrics.
- Row-level cleaning signals: Columns such as
dropandduplicatemark items recommended for exclusion or linking to canonical duplicates. So what: training recipes can use only the non-dropped, non-duplicate subset to match the authors' recommended pre-training split or intentionally include noisy rows to study robustness. - Recipe-driven design: The dataset was assembled to mirror the mGTE technical report / LateOn data recipe (Zhang et al., 2024) while making the raw, per-source inputs available. So what: teams building single- or multi-vector embedding models can re-run or extend experiments without reconstructing the corpus composition from scratch.
Who It's For & Trade-offs
Great fit if you are training or researching text embedding/retrieval models and need a reproducible, componentized corpus to run ablations, replicate LateOn/mGTE-style experiments, or study source-mix effects on retrieval metrics. Look elsewhere if you need a single permissively-licensed monolithic dataset ready for immediate commercial redistribution: this collection preserves each source's original license, so you must verify license terms per source before using data in production.
Where It Fits
Use this dataset when experimental control over data composition matters (e.g., model comparisons, data‑centric ablations, and cleaning pipelines). It sits between raw web crawls and fully packaged commercial training corpora: more curated and source-attributed than raw dumps, but not relicensed or harmonized into a single permissive bundle.
Practical notes
- Recommended usage: filter to rows where
dropis False andduplicateis null for the clean pre-training subset (the dataset card provides an example SQL snippet). - Licensing: each source keeps its original license; the dataset does not relicense aggregated content — verify permissions for your use case. - Reproducibility: because the dataset is split into configs, you can script exact training mixtures to reproduce published LateOn/GTE experiments or iterate on new mixtures.
