Why this matters
At web scale, many engineering trade‑offs (index size vs recall, quantization loss, sharding cost, ANN accuracy) only become visible with billions of vectors. This release supplies a reproducible 10.07B‑document slice of FineWeb with paired dense and sparse embeddings and exact brute‑force top‑1000 ground truth computed over the full corpus — letting researchers measure those trade‑offs without approximating the reference results.
What Sets It Apart
- True web‑scale corpus: 10,074,324,060 FineWeb documents (English slice) with original text, URL, crawl dump id and timestamp — enabling realistic indexing and filtering experiments. This corpus originates from FineWeb, itself derived from Common Crawl and released under ODC‑BY‑1.0.
- Paired dense + sparse embeddings: 768‑dim unit‑norm dense vectors and mGTE sparse weights (vocabulary size 250,048). The pairing supports pure dense, pure sparse, and hybrid retrieval comparisons under a single dataset.
- Exact brute‑force ground truth at scale: ~119,953 MS MARCO queries (100k dense, 10k sparse, ~5k text‑filtered, 5k structured‑filtered) with exact top‑1000 results computed using GPU brute‑force (Supernova nova‑bf), avoiding approximations that can mask small but meaningful algorithmic differences.
- Production‑scale footprint: vector data and source text are massive (Qdrant reports ~24.47 TiB of vectors and ~28.66 TiB of text/metadata for the full release), so the dataset is realistic for measuring index build cost, memory footprint, and sharding strategies.
Who it’s for and trade‑offs
Great fit if you need rigorous, large‑scale evaluation of retrieval systems — for example, comparing ANN algorithms, quantization schemes, sparse/dense fusion, or measuring index build time/cost across realistic sharding setups. The paired embeddings and filterable queries make it suitable for research on filtered retrieval and hybrid ranking.
Look elsewhere if you need small, easy‑to‑download examples for rapid prototyping: the full corpus is enormous and impractical to host locally without significant storage and compute. Also note licensing constraints: the assembled dataset is ODC‑BY‑1.0, while MS MARCO query text and regenerated query embeddings remain subject to MS MARCO’s non‑commercial research license — check those terms before downstream use.
Where it fits
Use this dataset when approximate benchmarks (1M scale) no longer expose relevant system behaviors and you require ground‑truth comparisons at 10B scale. It’s complementary to smaller retrieval benchmarks (e.g., MS MARCO alone) and to embedding corpora that provide only dense vectors: here you get both dense and token‑level sparse representations derived from gte‑multilingual‑base, plus real web metadata for filtered experiments.
