srush/LLM-Training-Puzzles

A hands-on puzzle collection exploring distributed training primitives across thousands of GPUs, covering memory efficiency and compute pipelining strategies critical to large-scale model training. Each puzzle presents concrete challenges in multi-GPU coordination, requiring implementation of techniques like gradient accumulation, pipeline parallelism, and communication optimization. Designed for Colab execution with progressive difficulty, building on prior puzzle series covering GPU kernels, tensors, autodiff, and transformers.

1,157 stars. No commits in the last 6 months.