Training frontier-scale models on domestic accelerators is not just a hardware substitution: it changes how model architecture, memory use, communication, and host scheduling must be designed together. This report’s central contribution is a systems account of that co-design, grounded in a 285B-parameter Xing 4 pretraining run on 8,192 Ascend 910C cards, rather than a direct benchmark comparison with other frontier models.
Key Findings
- The report studies sparse activation, attention, model depth and width, and speculative decoding as coupled design choices. The open-sourced 29B model activates 4B parameters per token and natively supports 256K-token contexts.
- Its pretraining stack uses MindSpore HyperParallel for topology-aware hybrid sharding, distributed optimizer execution, and long-sequence memory management; CPU affinity and core isolation help reduce host-side scheduling interference.
- The larger 285B configuration is reported to activate about 17B parameters per token and support contexts up to one million tokens. Reported system measurements include up to 4.68× prefill throughput over an MLA-only baseline and 2.67×–3.77× decoding throughput from speculative decoding.
- TeleGym connects agent execution, evaluation, trajectory construction, and policy optimization. Asynchronous execution and partial rollouts address tasks with different durations while retaining trajectory continuity across policy updates.
Who Should Read It
A good fit for researchers and infrastructure teams evaluating large-model training on Ascend, MoE and long-context design, or agent post-training systems. Look elsewhere for a broad head-to-head assessment of model quality: the report focuses on architecture and systems engineering, and its throughput figures are tied to specified system baselines and configurations.