Before Megatron-LM, scaling a transformer past a single GPU's memory meant hand-rolling the math of splitting attention and MLP layers across devices. Its real contribution was packaging that splitting into a few lines of code change, turning model parallelism from a research stunt into a reusable primitive that the rest of the field now builds on.
Key Capabilities
- Stacks five parallelism axes — tensor, pipeline, data, context, and expert — so the same codebase trains anything from a 2B model on a few cards to a 462B model on thousands of GPUs.
- Hits roughly 47% Model FLOP Utilization on H100 clusters and demonstrates superlinear scaling past 6,000 GPUs, which is the difference between a training run costing weeks versus months.
- Splits into two layers: Megatron Core gives composable building blocks for teams writing their own framework, while Megatron-LM ships opinionated reference scripts for those who just want to train.
- Adds the production glue large runs actually need — FP8/FP4 mixed precision, activation checkpointing, distributed optimizers, and fault-tolerant distributed checkpointing with automatic restart.
Who It's For
Great fit if you are pre-training or heavily fine-tuning models too large for a single node and want battle-tested parallelism rather than reinventing it; NVIDIA itself and most large-model labs lean on these kernels. Look elsewhere if you are working at the scale of one or two GPUs, fine-tuning with LoRA, or want a high-level trainer — the abstractions and NVIDIA-GPU assumptions are overhead you will not recoup.
