AIAny
Icon for item

FinePDFs

Provides a reproducible, deduplicated corpus of text extracted from PDFs for LLM pretraining—about 3 trillion tokens from ~475 million documents in 1733 language-script pairs. Includes OCR and text extraction pipelines, per-page language IDs, MinHash deduplication, and is released under ODC‑By 1.0.

Introduction

Oops! Something went wrong

[next-mdx-remote-client] error compiling MDX: Unexpected character `2` (U+0032) before name, expected a character that can start a name, such as a letter, `$`, or `_` More information: https://mdxjs.com/docs/troubleshooting-mdx

Information

  • Websitehuggingface.co
  • OrganizationsHugging Face
  • AuthorsHynek Kydliček, Guilherme Penedo, Leandro von Werra
  • Published date2025/09/05

Categories

More Items

Hugging Face

Provides 10 million synchronized egocentric experience episodes with structured 3D/4D multimodal annotations — 2.88B RGB frames, 720M depth frames, 576M pose/mocap frames and ~1PB total. Designed for embodied AI, robotics, and multimodal pretraining; research-only, gated access.

Hugging Face

Provides OCR full text for 11.55M public-domain German newspaper pages (1638–1964) with per-page IIIF scans and ALTO XML coordinates; suited for historical NLP, language-model training, and OCR research. Pages carry explicit per-page public-domain licenses.

Hugging Face

Provides 545,431 math problems with model-generated solution traces (chain-of-thought and Python tool-integrated reasoning) verified against reference answers for training and evaluating LLM mathematical reasoning. Parquet-format dataset; DeepSeek‑V4‑Pro generated traces and mixed CC BY / CC BY‑SA licensing.