Most autoregressive LLM pretraining focuses only on next-token prediction. NCP-ArchPreview's core insight is to force the model to also predict higher-level discrete "concepts"—compact latent units that summarize several tokens—so the model learns semantic continuations at a coarser timescale and uses those predictions to guide token-level generation.
Key Findings
- Joint NTP (next-token) + NCP (next-concept) speeds effective pretraining: Stage 1 reaches the final training loss of an OLMo-3-7B baseline after consuming only 51.3% of the token budget, implying ~1.95× token-budget convergence.
- Strong downstream gains after full pretraining: +2.45 points macro-average vs OLMo-3-7B, including +5.99 on GSM8K and +4.28 on HumanEval, indicating improved reasoning and code capabilities.
- Architecture & efficiency: scaled to ~8.94B parameters, trained on 5.73T Dolma-3 tokens; using ~85% of standard computation approaches the loss of a parameter-aligned 8.9B baseline.
- Latent-space utility after pretraining: concept codebooks are small (~17M parameters) and can be updated for lightweight domain adaptation; injecting concept representations into a DFlash2 drafter increased mean accepted length by 4.17% with negligible overhead.
What Sets It Apart
- Explicit concept-level supervision: constructs concept vectors by mean-pooling groups of four token encoder states and predicts future concepts at one-quarter the token rate, creating a concrete training signal above token-level statistics.
- Product-quantized concept vocabulary: 32 codebooks provide a discrete yet differentiable concept space; predictions are differentiable weighted combinations of codewords, enabling end-to-end learning with standard autoregressive decoding.
- Modular design for adaptation: token backbone (encoder/decoder) remains usable while a small VQ module can be re-trained for domain shifts, enabling cheaper adaptation than full model fine-tuning.
Who it's for — tradeoffs
Great fit if you want to reduce pretraining token/computation budgets while improving reasoning/code benchmarks, or if you need a small, updatable interface (the ≈17M-parameter VQ module) for domain adaptation. Look elsewhere if you require strictly minimal system complexity (adds a concept module, codebooks, and altered training pipeline) or if your deployment stack cannot accommodate the added latent-interface or slightly different generation pathway.
Method details
- Pipeline: 16-layer token encoder → mean-pool every 4 token states into continuous concept vectors → 8-layer Concept Module predicts next concept → product quantization (32 codebooks) produces discrete concept vocab → 16-layer token decoder conditions on causally aligned concept predictions to produce tokens.
- Reported metrics: Stage 1 Overall AVG 49.04 vs 46.59 for OLMo-3-7B; MMLU 64.80 (+2.58); GSM8K 45.26 (+5.99); HumanEval 31.38 (+4.28). Context length used in released checkpoints: 8192 tokens.