NVIDIA/Megatron-LM
Ongoing research training transformer models at scale
Provides composable GPU-optimized building blocks for transformer training, including advanced parallelism strategies (tensor, pipeline, expert, and context parallelism), mixed precision support (FP16, BF16, FP8, FP4), and custom training pipeline construction. Achieves up to 47% Model FLOP Utilization on H100 clusters while scaling from 2B to 462B parameter models. Integrates with Hugging Face via Megatron Bridge for checkpoint conversion and works with NVIDIA NeMo framework for production-ready recipes.
15,633 stars. Actively maintained with 226 commits in the last 30 days.
Stars
15,633
Forks
3,689
Language
Python
License
—
Category
Last pushed
Mar 13, 2026
Commits (30d)
226
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/NVIDIA/Megatron-LM"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related models
huggingface/optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers...
openvinotoolkit/nncf
Neural Network Compression Framework for enhanced OpenVINOâ„¢ inference
huggingface/optimum-intel
🤗 Optimum Intel: Accelerate inference with Intel optimization tools
RBLN-SW/optimum-rbln
âš¡ A seamless integration of HuggingFace Transformers & Diffusers with RBLN SDK for efficient...
eole-nlp/eole
Open language modeling toolkit based on PyTorch