The shortage of high-quality, algorithmically meaningful code in public corpora makes scaling code-specialized LLM capabilities by raw data volume increasingly inefficient. UltraData-Stack-v3-L2 addresses this by applying a language-adaptive, file-level selection pipeline to The Stack v3, producing a much smaller but algorithmically denser training pool designed to amplify code-generation and problem-solving skills in downstream models.
What Sets It Apart
- Focused selection, not expansion: instead of enlarging the corpus, it retains ~20.10% of cleaned candidate files (127,133,027 files) yielding an estimated 245.50B tokens concentrated on algorithmic and problem-solving code. This preserves language breadth (11 languages) while boosting signal per token for coding tasks.
- Quantified training impact: under a controlled 10B-token continual-pretraining regime on a 1B base model, the dataset improved average pass@1 to 30.00 on EvalPlus (vs 25.37 for Stack-v3-train) and 22.75 on MultiPL-E (vs 17.30), demonstrating measurable gains from selection.
- Reusable, classifier-driven pipeline: selection uses UltraData-Code-L2-Classifier with Qwen3-Embedding-0.6B file embeddings and language-specific thresholds, enabling transfer of the L2 objective to alternative source corpora.
Who it's for and tradeoffs
Great fit if you want higher algorithmic density per training token for code-model pretraining or mid-training, or need a multilingual, license-traceable code corpus optimized for coding benchmarks. Look elsewhere if you require exhaustive archival provenance of every repository file (L0-style archival) or maximal raw coverage—this dataset intentionally excludes many non-algorithmic files and retains only files passing quality and relevance constraints. Also note the dataset inherits upstream license obligations (ODC-By and original repo licenses).
