This snapshot packages arXiv's metadata, version history and submission files into queryable Parquet configs so researchers can work at scale without rebuilding crawlers. Its key value is offering both assembled TeX text rows and per-file content indexes (source/latex/pdf/ps) so you can choose lightweight metadata queries or pull large raw assets when needed.
What Sets It Apart
- Multi-granular Parquet configs: small metadata and versions indexes (1.6 GB) plus heavy-content configs (
paper_text,latex,source,pdf,ps) that let you fetch exactly what you need. This makes selective streaming and DuckDB/Arrow queries practical compared with raw tar archives. - Full provenance and verification: every Parquet file is checksummed, content digests are provided, and detailed coverage statistics and gap analyses are published (snapshot dates, counts, PDF provenance). This helps reproducible dataset building and legal/license checks.
- Rich assembly insights:
paper_textsupplies one assembled TeX string for ~2.86M papers (retaining macros/comments), with token estimates (~78–81B tokens using cl100k_base) and explicit decoding/quality caveats documented so users can prefilter before model training.
Who It's For and Tradeoffs
Great fit if you need a comprehensive, audited mirror of arXiv for model pretraining, retrieval index construction, large-scale metadata research or reproducible experiments. Use the small index configs for filtering and only download heavy content on demand. Look elsewhere if you require a continually updating live mirror (this is a one-off snapshot) or prefer already-extracted plain-text PDFs — paper_text retains LaTeX syntax and some templates/withdrawal stubs, so it needs cleaning for direct language-model training.
