LLM Implementation From Scratch Transformer Models

Educational repositories focused on building Large Language Models from first principles using PyTorch, emphasizing step-by-step understanding of transformer architecture, tokenization, and training mechanics. Does NOT include fine-tuning existing models, inference optimization, or production deployment frameworks.

There are 44 llm implementation from scratch models tracked. 1 score above 50 (established tier). The highest-rated is rasbt/LLMs-from-scratch at 69/100 with 87,892 stars. 1 of the top 10 are actively maintained.

Get all 44 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=llm-implementation-from-scratch&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	rasbt/LLMs-from-scratch Implement a ChatGPT-like LLM in PyTorch from scratch, step by step	69	Established	87,892	Jupyter Notebook
2	datawhalechina/llms-from-scratch-cn 仅需Python基础，从0构建大语言模型；从0逐步构建GLM4\Llama3\RWKV6，深入理解大模型原理	48	Emerging	4,010	Jupyter Notebook
3	FareedKhan-dev/train-llm-from-scratch A straightforward method for training your LLM, from downloading data to...	45	Emerging	531	Jupyter Notebook
4	facebookresearch/LayerSkip Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative...	45	Emerging	361	Python
5	kmeng01/rome Locating and editing factual associations in GPT (NeurIPS 2022)	44	Emerging	737	Python
6	analyticalrohit/llms-from-scratch Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.	43	Emerging	26	Jupyter Notebook
7	codewithdark-git/Building-LLMs-from-scratch This repository guides you through the process of building a GPT-style Large...	41	Emerging	51	Jupyter Notebook
8	geeks-of-data/knowledge-gpt Extract knowledge from all information sources using gpt and other language...	40	Emerging	291	Python
9	huangwl18/language-planner Official Code for "Language Models as Zero-Shot Planners: Extracting...	36	Emerging	278	Jupyter Notebook
10	skyloevil/llm-scratch-pytorch lm-scratch-pytorch - The code is designed to be beginner-friendly, with a...	35	Emerging	100	Jupyter Notebook
11	therealoliver/Deepdive-llama3-from-scratch Achieve the llama3 inference step-by-step, grasp the core concepts, master...	35	Emerging	626	Jupyter Notebook
12	clabrugere/scratch-llm Implements a LLM similar to Meta's Llama 2 from the ground up in PyTorch,...	33	Emerging	38	Python
13	OpenSparseLLMs/LLaMA-MoE-v2 🚀 LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of...	33	Emerging	93	Python
14	HxCodeWarrior/StellarByte 从零实现基础的Transformer的Decoerder-Only模型，并进行模型升级，构建专属于自己的LLM模型	32	Emerging	6	Python
15	FareedKhan-dev/create-million-parameter-llm-from-scratch Building a 2.3M-parameter LLM from scratch with LLaMA 1 architecture.	32	Emerging	201	Jupyter Notebook
16	zhanshijinwat/Steel-LLM Train a 1B LLM with 1T tokens from scratch by personal	31	Emerging	791	Jupyter Notebook
17	Mmorgan-ML/Phase-Slip-Sampler Phase-Slip is a stochastic intervention architecture that operates on the...	31	Emerging	6	Python
18	joelbarmettlerUZH/ConceptFormer Towards Finding the Essence of Everything in Large Language Models	30	Emerging	13	Python
19	ai-art-dev99/llm-from-scratch Build a Large Language Model From Scratch	29	Experimental	22	Jupyter Notebook
20	vipulraheja/iterater Official implementation of the paper "IteraTeR: Understanding Iterative...	28	Experimental	80	Python
21	bloomberg/minilmv2.bb Our open source implementation of MiniLMv2...	26	Experimental	61	Python
22	UCSB-NLP-Chang/ULD Implementation of paper 'Reversing the Forget-Retain Objectives: An...	26	Experimental	26	Python
23	jpwahle/emnlp23-paraphrase-types The official implementation of the EMNLP 2023 paper "Paraphrase Types for...	25	Experimental	12	Python
24	NamrataThakur/Large_Language_Model_From_Scratch_Implementation Implementing an LLM from scratch block-by-block using PyTorch	24	Experimental	—	Jupyter Notebook
25	Swamy-s-Tech-Skills-Academy-2026/llms-from-scratch-practice Hands-on learning repository for building a GPT-style Large Language Model...	22	Experimental	—	Jupyter Notebook
26	nishantb06/smolLM Reverse Engineering SmolLM2 model and training it from scratch	20	Experimental	1	Python
27	newfull5/NLLB-200-Distilled-350M-en-ko nllb-200 distilled 350M for English to Korean translation	20	Experimental	28	Jupyter Notebook
28	Arlchoose-code/Indonesian-LLM-Starter A starter kit for building your own Indonesian Large Language Model (LLM)...	20	Experimental	1	Python
29	bijinc/speculoos efficient speculative sampling for language models	19	Experimental	—	Python
30	rafaelvp-db/db-ancient-code-translation Simple repo showing code-to-code and code-to-text capabilities using LLMs on...	18	Experimental	5	Python
31	shreyansh26/LLM-Sampling A collection of various LLM sampling methods implemented in pure Pytorch	17	Experimental	28	Python
32	wasim/scaling-specialization-dense-lms Do dense LMs develop MoE-like specialization as they scale? Measure it,...	16	Experimental	1	Python
33	bassrehab/speculative-decoding Reference implementation of LLM inference acceleration techniques. Includes...	16	Experimental	1	Python
34	NaS-Research/knowledge-model Our knowledge system systematically ingests, processes, and indexes...	16	Experimental	1	Python
35	SoelMgd/Poker_Transformers LLMs trained for Poker	14	Experimental	9	Jupyter Notebook
36	ghassenov/llm_from_scratch A GPT-2 model from scratch built to explore the inner workings of...	14	Experimental	4	Jupyter Notebook
37	adarsh-crafts/llama-llm-from-scratch Educational, from-scratch implementation of a LLaMA-style LLM using PyTorch...	14	Experimental	4	Jupyter Notebook
38	ldr7/language_model_from_scratch Build a language model from scratch.	14	Experimental	1	Jupyter Notebook
39	YUGESHKARAN/Clash_of_Clans_Language_Model A mini language model from scratch using PyTorch, with approximately 2.96...	11	Experimental	—	Jupyter Notebook
40	harishm17/build-llm-from-scratch From‑scratch LLM notebooks: Transformers, BPE tokenizer, PyTorch...	11	Experimental	—	Jupyter Notebook
41	theosorus/French-Language-Model In this project, I built a French Large Language Model only with pytorch	11	Experimental	7	Python
42	RobinSmits/Schaapje Schaapje - A Dutch Small Language Model	11	Experimental	2	Jupyter Notebook
43	mohitpg/LLMs-from-scratch A collection of LLMs implemented from scratch using pytorch	10	Experimental	1	Python
44	VisualJoyce/TERepo [ACL 2023] A Text Editing Repository for reproduction and innovation.	10	Experimental	1	Python

Comparisons in this category