Lightning-AI/litgpt

Implements models from scratch without abstraction layers and combines Flash Attention with FSDP for distributed training across 1-1000+ GPUs/TPUs. Supports parameter-efficient finetuning via LoRA/QLoRA with mixed-precision quantization (fp4/8/16/32) to reduce GPU memory requirements, while integrating with PyTorch Lightning and Lightning Cloud infrastructure for end-to-end pretraining, finetuning, and deployment workflows through declarative YAML recipes.

13,225 stars and 15,196 monthly downloads. Actively maintained with 5 commits in the last 30 days. Available on PyPI.