Megatron-LM

NVIDIA’s model-parallel training library for GPT-like transformers at multi-billion-parameter scale.

Visit Website

Introduction

Back

Information

Websitedeveloper.nvidia.com
AuthorsNVIDIA
Published date2019/09/17

More Items

Ray

2017

RISELab (UC Berkeley), Anyscale Inc.

Ray is an open-source distributed compute engine that lets you scale Python and AI workloads—from data processing to model training and serving—without deep distributed-systems expertise.

ai-development ai-framework ai-train ai-serving

FastChat

2023

Large Model Systems (LMSYS)

An open platform for training, serving and evaluating chat-oriented LLMs—powering Vicuna & Chatbot Arena.

ai-development ai-framework ai-train

LLaMA-Factory

2024

hiyouga

Zero-code CLI & WebUI to fine-tune 100+ LLMs/VLMs with LoRA, QLoRA, PPO, DPO and more.

ai-development ai-framework ai-train

Overview

Megatron-LM pioneered tensor and pipeline model parallelism, enabling training of GPT-style models up to hundreds of billions of parameters with high GPU efficiency.

Key Capabilities

Tensor & pipeline parallel APIs with minimal code changes
Fused layer-norm, bias-gelu and FlashAttention kernels
Activation recomputation & distributed optimizer sharding
Megatron Core library for plug-and-play integration
Extensive examples and Docker images for quick start