Llm Knowledge Distillation Transformer Models

There are 28 llm knowledge distillation models tracked. 1 score above 50 (established tier). The highest-rated is scaleapi/llm-engine at 56/100 with 821 stars. 2 of the top 10 are actively maintained.

Get all 28 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=llm-knowledge-distillation&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	scaleapi/llm-engine Scale LLM Engine public repository	56	Established	821	Python
2	modelscope/easydistill a toolkit on knowledge distillation for large language models	47	Emerging	292	Python
3	AGI-Arena/MARS The official implementation of MARS: Unleashing the Power of Variance...	47	Emerging	716	Python
4	Wang-ML-Lab/bayesian-peft Bayesian Low-Rank Adaptation of LLMs: BLoB [NeurIPS 2024] and TFB [NeurIPS 2025]	38	Emerging	35	Python
5	AGI-Edgerunners/LLM-Adapters Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for...	38	Emerging	1,229	Python
6	yifanzhang-pro/HLA Official Project Page for HLA: Higher-order Linear Attention...	36	Emerging	45	HTML
7	sangmichaelxie/doremi Pytorch implementation of DoReMi, a method for optimizing the data mixture...	35	Emerging	352	HTML
8	ZO-Bench/ZO-LLM [ICML‘24] Official code for the paper "Revisiting Zeroth-Order Optimization...	35	Emerging	124	Python
9	Liuhong99/Sophia The official implementation of “Sophia: A Scalable Stochastic Second-order...	34	Emerging	985	Python
10	yang-ai-lab/OSF-Open-Sleep-FM OSF: On Pre-training and Scaling of Sleep Foundation Models	34	Emerging	10	Jupyter Notebook
11	ShiZhengyan/DePT [ICLR 2024] This is the repository for the paper titled "DePT: Decomposed...	33	Emerging	102	Python
12	YJiangcm/Lion [EMNLP 2023] Lion: Adversarial Distillation of Proprietary Large Language Models	32	Emerging	212	Python
13	golololologol/LLM-Distillery A pipeline for LLM knowledge distillation	32	Emerging	112	Python
14	shufangxun/LLaVA-MoD [ICLR 2025] LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation	31	Emerging	223	Python
15	horus-ai-labs/DistillFlow Library for model distillation	30	Emerging	165	Python
16	OatmealLiu/FineR [ICLR'24] Democratizing Fine-grained Visual Recognition with Large Language Models	28	Experimental	190	Python
17	Tebmer/Awesome-Knowledge-Distillation-of-LLMs This repository collects papers for "A Survey on Knowledge Distillation of...	27	Experimental	1,264	—
18	Qwen-Applications/STAR STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function...	26	Experimental	39	Python
19	ROIM1998/APT [ICML'24 Oral] APT: Adaptive Pruning and Tuning Pretrained Language Models...	26	Experimental	47	Python
20	pittisl/GreenTrainer Code for paper "Towards Green AI in Fine-tuning Large Language Models via...	25	Experimental	13	Python
21	iboing/CorDA CorDA: Context-Oriented Decomposition Adaptation of Large Language Models...	24	Experimental	55	Python
22	pdaicode/awesome-LLMs-finetuning Collection of resources for finetuning Large Language Models (LLMs).	24	Experimental	113	—
23	teilomillet/retrain a Python library that uses Reinforcement Learning (RL) to train LLMs.	22	Experimental	42	Python
24	TamSiuhin/OPPU Official Implementation of "Democratizing Large Language Models via...	21	Experimental	44	Python
25	wshi83/MedAdapter [EMNLP'24] MedAdapter: Efficient Test-Time Adaptation of Large Language...	16	Experimental	36	Python
26	amazon-science/mezo_svrg Code the ICML 2024 paper: "Variance-reduced Zeroth-Order Methods for...	14	Experimental	12	Python
27	BaohaoLiao/mefts [NeurIPS 2023] Make Your Pre-trained Model Reversible: From Parameter to...	12	Experimental	33	Python
28	kriskrisliu/PAT [AAAI 2025] PAT: Pruning-Aware Tuning for Large Language Models	11	Experimental	36	Python