LLMs-from-scratch and scratch-llm
These are complements rather than competitors: the first provides a comprehensive, production-oriented pedagogical framework for building transformer-based LLMs (covering architecture, training, and inference), while the second offers a lightweight, ground-up implementation specifically focused on replicating Llama 2's design for educational purposes, allowing learners to study both a general approach and a specific modern architecture.
About LLMs-from-scratch
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Covers the complete pipeline from tokenization and attention mechanisms through pretraining on unlabeled data and finetuning for classification and instruction-following tasks. Includes practical implementations of multi-head attention, causal masking, and parameter-efficient techniques like LoRA, alongside code for loading pretrained model weights. Organized as Jupyter notebooks and standalone Python scripts that progressively build a functional GPT architecture while explaining each component's role in modern LLM training.
About scratch-llm
clabrugere/scratch-llm
Implements a LLM similar to Meta's Llama 2 from the ground up in PyTorch, for educational purposes.
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work