Most large code corpora mix boilerplate, tests, generated files, and genuinely algorithmic implementations — this noise dilutes signal during pretraining. By organizing repository archives into a tiered pipeline and emitting both selected natural code (L2) and task-oriented synthetic exercises (L3), the dataset isolates algorithmic intent and supplies explicit generation supervision that improves code-model capabilities per token.
What Sets It Apart
- Tiered, provenance-preserving pipeline: starts from a full L0 archive of ~192M public GitHub repositories and applies language-specific cleaning, near-duplicate deduplication, role prediction, and quality filtering to produce L1/L2 artifacts with traceability. This keeps selection decisions auditable and reproducible.
- Algorithmic selection (L2): selects ~400B tokens of files deemed algorithmically relevant using file-role supervision, heuristic language cues, and embedding-based filters. Each file includes metadata such as predicted role, algo_rel_score, and quality_score for informed sampling or filtering.
- Implementation-grounded synthesis (L3): converts selected implementations into standalone programming exercises (task, analysis, solution, tests), yielding ~150B tokens that provide explicit paired supervision for code generation and evaluation.
- Empirical gains in controlled experiments: L2 vs L1 and L2+L3 mixtures consistently improve pass@1 on EvalPlus and MultiPL-E benchmarks in 10B-token controlled trainings, showing selection and synthesis deliver measurable per-token benefits.
Who it's for and tradeoffs
Great fit if you need high-quality pretraining or continual-pretraining code corpora with clear provenance, per-file metadata for selection, or synthetic task records for supervised fine-tuning and evaluation. It’s valuable for research comparing selection/synthesis strategies, building code-focused LLMs, or generating benchmarks aligned with implementation intent.
Look elsewhere if you need permissive, unconditional redistribution of original repository content (source repo licenses still apply), or if your use case requires curated domain-specific private code — the dataset derives from public GitHub sources and enforces license and takedown constraints. The dataset is large and designed for pretraining workflows rather than lightweight on-device inference datasets.
