Large-scale, curriculum-structured training mixtures matter when trying to push compact models toward stronger mathematical reasoning and agentic behavior while keeping the development pipeline reproducible. This dataset is the per-stage data suite used in ZGCM-1’s lifecycle and exposes token counts, quality tiers, source families, and a shared Parquet schema so researchers can study mixtures, long-context curricula, and SFT composition without rebuilding the entire pipeline.
What Sets It Apart
- Explicit multi-stage splits and scale: the collection documents pretraining (
4.19T tokens split across a curriculum stage and a full-mixture stage), midtraining (≈600B tokens across 16K/64K/256K target contexts), and SFT (~4.57M samples / 19.46B packed tokens). This makes it straightforward to reproduce stagewise experiments. - Long-context and agentic focus: midtraining includes long-context configurations (16K→64K→256K) and dedicated agentic/trajectory records reformatted as interaction-state transitions to support agentic supervision and tool-use traces.
- Rich, provenance-aware schema: records are stored as Zstandard-compressed Parquet with a shared schema (text, source, source_group, upstream_repos, licenses, release_mode, category, token_count, metadata_json), enabling per-sample license and upstream-repo tracking.
- Broad source families and task coverage: combines web text, FinePDF/OCR, academic papers, code (GitHub-derived indices), math/LaTeX corpora, QA, reasoning, and encyclopedic content, intended for pretraining, capability midtraining, and SFT-oriented tasks.
Who it's for & tradeoffs
Great fit if you need a reproducible, stagewise training mixture for LLM research—especially experiments on long-context scaling, curriculum effects, or SFT strategies for reasoning, code, and agentic behaviors. It is less suitable if you need only small curated benchmarks or fully permissive licensing: the collection aggregates many upstream licenses (including CC-BY-NC variants and index-only records), so legal review is required for commercial use. Preparing and training on these files also demands substantial storage and compute resources.
Where it fits
Use this data suite when you want an open, documented training pipeline that mirrors a contemporary research-grade pretrain→midtrain→SFT lifecycle, rather than one-off benchmark datasets. It complements Common Crawl-style web dumps with extra curated math, code, and agentic traces and emphasizes long-context examples and provenance.
