Why this matters
Public, self-contained corpora that capture repository context are rare at the scale needed to pretrain competitive code LLMs. The Stack v3 train split delivers a ready-to-use, repository-grouped corpus with file contents included so researchers and engineers can experiment with full-repo context without rebuilding large-scale crawlers or dedup pipelines.
What Sets It Apart
- Repository grouping with inline contents: every row is a full repository and files[].content is the decoded UTF-8 source text, so models can be trained or evaluated with repository structure and file adjacency preserved. This avoids piecemeal reassembly from separate blobs.
- Near-dedup and quality filtering tuned for pretraining: language-agnostic MinHash-LSH clustering retains one representative per duplicate cluster (representative chosen by stars/forks/license/age), plus heuristics to drop low-quality, oversized or generated files and PII redaction applied to detected entities.
- Scale and language breadth: the train split contains ~15.9 TB of code (≈4.9 trillion tokens) across 713 programming/markup languages and ~173M repository rows, reflecting GitHub state as of 2025-08-07.
Key details for practitioners
- Format and access: released as Parquet/Arrow shards; usable via the Hugging Face datasets library (streaming supported) or queried directly with DuckDB/Polars via the HF filesystem. Each repo row includes repo metadata (stars, forks, commit_id) and per-file fields (language via go-enry, detected_licenses via ScanCode, is_vendor, size_bytes, file_timestamp).
- Licensing and provenance: dataset published under ODC-By v1.0; file-level detected licenses are provided and non-permissive files were excluded. Users must still respect original file licenses when reusing code and follow attribution requirements as applicable.
- Safety and limitations: PII was redacted with StarPII heuristics but detection is imperfect and some sensitive data may remain. The corpus may contain malicious code and biased language distributions (popular languages dominate). The snapshot captures only default-branch HEADs and omits git history.
Who it fits and trade-offs
Great fit if you need a large, reproducible, repo-context corpus to pretrain or benchmark code LLMs and can operate at multi-terabyte scale (or stream shards). Choose stack-v3-full instead if you need every near-duplicate, finer-grained content partitioning, or want to re-run custom dedup/licensing filters. Look elsewhere if you require curated, human-reviewed code examples, full commit histories, or guarantees that all PII has been removed.
