Vision Transformer Optimization ML Frameworks

Official implementations and research papers focused on improving Vision Transformer architectures through efficiency enhancements, dynamic token pruning, hierarchical designs, and architectural innovations. Does NOT include general computer vision frameworks, multimodal models, or non-transformer-based vision approaches.

There are 109 vision transformer optimization frameworks tracked. 8 score above 50 (established tier). The highest-rated is zhanghang1989/ResNeSt at 67/100 with 3,264 stars and 11,896 monthly downloads. 1 of the top 10 are actively maintained.

Get all 109 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=ml-frameworks&subcategory=vision-transformer-optimization&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Framework	Score	Tier	Stars	Language
1	zhanghang1989/ResNeSt ResNeSt: Split-Attention Networks	67	Established	3,264	Python
2	berniwal/swin-transformer-pytorch Implementation of the Swin Transformer in PyTorch.	63	Established	859	Python
3	Jittor/jittor Jittor is a high-performance deep learning framework based on JIT compiling...	59	Established	3,221	Python
4	NVlabs/FasterViT [ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision...	54	Established	907	Python
5	ViTAE-Transformer/ViTPose The official repo for [NeurIPS'22] "ViTPose: Simple Vision Transformer...	53	Established	1,957	Python
6	sniklaus/pytorch-pwc a reimplementation of PWC-Net in PyTorch that matches the official Caffe version	51	Established	654	Python
7	microsoft/CvT This is an official implementation of CvT: Introducing Convolutions to...	51	Established	602	Python
8	gaohuang/MSDNet Multi-Scale Dense Networks for Resource Efficient Image Classification （ICLR...	50	Established	461	Lua
9	vra/dinov2-retrieval A cli program of image retrieval using dinov2	49	Emerging	79	Python
10	tobna/WhatTransformerToFavor Github repository for the paper Which Transformer to Favor: A Comparative...	49	Emerging	33	Python
11	Khrylx/AgentFormer [ICCV 2021] Official PyTorch Implementation of "AgentFormer: Agent-Aware...	49	Emerging	309	Python
12	google-research/big_transfer Official repository for the "Big Transfer (BiT): General Visual...	47	Emerging	1,538	Python
13	richzhang/PerceptualSimilarity LPIPS metric. pip install lpips	47	Emerging	4,185	Python
14	iduta/pyconv Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual...	46	Emerging	331	Python
15	jwr1995/dc1d A 1D implementation of a deformable convolutional layer in PyTorch with a few tricks.	45	Emerging	46	Python
16	walsvid/CoordConv Pytorch implementation of "An intriguing failing of convolutional neural...	45	Emerging	163	Jupyter Notebook
17	VicenteVivan/geo-clip This is an official PyTorch implementation of our NeurIPS 2023 paper...	45	Emerging	330	Python
18	bwconrad/vit-finetune Fine-tuning Vision Transformers on various classification datasets	45	Emerging	115	Python
19	raoyongming/DynamicViT [NeurIPS 2021] [T-PAMI] DynamicViT: Efficient Vision Transformers with...	45	Emerging	651	Jupyter Notebook
20	clovaai/rexnet Official Pytorch implementation of ReXNet (Rank eXpansion Network) with...	45	Emerging	451	Python
21	innat/DOLG-TensorFlow Implementation of Deep Orthogonal Fusion of Local and Global Features in TensorFlow 2	44	Emerging	26	Jupyter Notebook
22	Yangzhangcst/Transformer-in-Computer-Vision A paper list of some recent Transformer-based CV works.	44	Emerging	1,435	—
23	LeapLabTHU/DAT Repository of Vision Transformer with Deformable Attention (CVPR2022) and...	44	Emerging	925	Python
24	kampta/DeepLayout PyTorch implementation of "LayoutTransformer: Layout Generation and...	44	Emerging	165	Python
25	ShirAmir/dino-vit-features Official implementation for the paper "Deep ViT Features as Dense Visual...	44	Emerging	464	Python
26	Renumics/mesh2vec Turn CAE mesh data => aggregated element feature vectors for ML	43	Emerging	15	KFramework
27	thuml/Xlearn Transfer Learning Library	43	Emerging	463	Jupyter Notebook
28	fkodom/yet-another-retnet A simple but robust PyTorch implementation of RetNet from "Retentive...	42	Emerging	106	Python
29	htdt/hyp_metric Hyperbolic Vision Transformers: Combining Improvements in Metric Learning \|...	42	Emerging	209	Python
30	chenhaoxing/SSFormers This repository is the code of the paper "Sparse Spatial Transformers for...	42	Emerging	49	Python
31	mit-han-lab/offsite-tuning Offsite-Tuning: Transfer Learning without Full Model	42	Emerging	387	Python
32	alon-albalak/TLiDB Transfer Learning in Dialogue Benchmarking Toolkit	41	Emerging	14	Python
33	ChristophReich1996/MaxViT PyTorch reimplementation of the paper "MaxViT: Multi-Axis Vision...	40	Emerging	164	Python
34	baraline/convst Implementation of the Random Dilated Shapelet Transform algorithm along with...	40	Emerging	35	Python
35	dongkyunk/DOLG-pytorch Unofficial PyTorch Implementation of "DOLG: Single-Stage Image Retrieval...	40	Emerging	135	Python
36	AaltoVision/DGC-Net A PyTorch implementation of "DGC-Net: Dense Geometric Correspondence Network"	39	Emerging	206	Jupyter Notebook
37	amazon-science/semi-vit PyTorch implementation of Semi-supervised Vision Transformers	39	Emerging	61	Python
38	NVlabs/FAN Official PyTorch implementation of Fully Attentional Networks	39	Emerging	480	Python
39	PracticumAI/transfer_learning Transfer learning is a powerful method allowing you to repurpose an AI model...	38	Emerging	3	Jupyter Notebook
40	DavidLandup0/deepvision PyTorch and TensorFlow/Keras image models with automatic weight conversions...	38	Emerging	42	Python
41	SunghwanHong/Cost-Aggregation-transformers Official implementation of CATs	37	Emerging	134	Python
42	daniel-code/TubeViT An unofficial implementation of TubeViT in "Rethinking Video ViTs: Sparse...	37	Emerging	94	Python
43	FrancescoSaverioZuppichini/ViT Implementing Vi(sion)T(transformer)	37	Emerging	453	—
44	bryanlimy/V1T [TMLR 2023] V1T: Large-scale mouse V1 response prediction using a Vision Transformer	37	Emerging	23	Jupyter Notebook
45	ViTAE-Transformer/ViTAE-Transformer The official repo for [NeurIPS'21] "ViTAE: Vision Transformer Advanced by...	35	Emerging	281	Python
46	YifanXu74/Evo-ViT Official implement of Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision...	35	Emerging	74	Python
47	GuanRunwei/Awesome-Vision-Transformer-Collection Variants of Vision Transformer and its downstream tasks	34	Emerging	257	—
48	MosbehBarhoumiRAI/VITON-PRE-PROCESSING This repository contains the initial implementation of pre-processing for...	34	Emerging	52	Jupyter Notebook
49	AnkurDeria/MFT Pytorch implementation of Multimodal Fusion Transformer for Remote Sensing...	33	Emerging	237	Jupyter Notebook
50	xiusu/ViTAS Code for ViTAS_Vision Transformer Architecture Search	33	Emerging	51	Python
51	intel/transfer-learning Libraries and tools to support Transfer Learning	33	Emerging	20	Python
52	graldij/transformer-fusion Official repository of the "Transformer Fusion with Optimal Transport"...	32	Emerging	31	Python
53	johndpope/OmniTransfer-hack OmniTransfer implementation for LTX-2 (work in progress)	32	Emerging	7	Python
54	paulgavrikov/CNN-Filter-DB A database of over 1.4 billion 3x3 convolution filters extracted from...	31	Emerging	34	Jupyter Notebook
55	shashankvkt/DoRA_ICLR24 This repo contains the official implementation of ICLR 2024 paper "Is...	31	Emerging	95	Python
56	apple/parameterized-transforms torchvision-based transforms that provide access to parameterization	30	Emerging	16	Python
57	nerminnuraydogan/vision-transformer Vision Transformer explanation and implementation with PyTorch	30	Emerging	67	Jupyter Notebook
58	altndrr/vic Code implementation of our NeurIPS 2023 paper: Vocabulary-free Image Classification	30	Emerging	107	Python
59	ViTAE-Transformer/ViTAE-VSA The official repo for [ECCV'22] "VSA: Learning Varied-Size Window Attention...	30	Emerging	158	Python
60	billpsomas/simpool This repo contains the official implementation of ICCV 2023 paper "Keep It...	29	Experimental	101	Python
61	NU-CUCIS/CrossPropertyTL Cross-property Deep Transfer Learning	29	Experimental	9	Jupyter Notebook
62	Rishit-dagli/Transformer-in-Transformer An Implementation of Transformer in Transformer in TensorFlow for image...	29	Experimental	43	Jupyter Notebook
63	mako443/Text2Pos-CVPR2022 Code, dataset and models for our CVPR 2022 publication "Text2Pos"	28	Experimental	54	Python
64	iduta/coconv [ICCV W] Contextual Convolutional Neural Networks...	28	Experimental	14	Python
65	pavlo-melnyk/mlgp-embedme The official implementation of the "Embed Me If You Can: A Geometric...	27	Experimental	9	Jupyter Notebook
66	JoanaR/multi-mode-CNN-pytorch A PyTorch implementation of the Multi-Mode CNN to reconstruct Chlorophyll-a...	27	Experimental	10	Jupyter Notebook
67	shikishima-TasakiLab/Involution-PyTorch Unofficial PyTorch reimplemention of the paper "Involution: Inverting the...	26	Experimental	21	C++
68	ViTAE-Transformer/LeMeViT The official repo for [IJCAI'24] "LeMeViT: Efficient Vision Transformer with...	26	Experimental	53	Python
69	materight/RepNet-pytorch A PyTorch port with pre-trained weights of RepNet, from "Counting Out Time:...	26	Experimental	40	Python
70	benbergner/cropr A token pruning method that accelerates ViTs for various tasks while...	25	Experimental	27	Python
71	altndrr/vicss Code implementation of our paper: Vocabulary-free Image Classification and...	25	Experimental	5	Python
72	dimiz51/FaceViT FaceViT: A multi-task Vision Transformer for face detection, age estimation,...	25	Experimental	4	Jupyter Notebook
73	insitro/ContextViT Contextual Vision Transformers for Robust Representation Learning	25	Experimental	15	Python
74	WalterSimoncini/fungivision Library implementation of "No Train, all Gain: Self-Supervised Gradients...	23	Experimental	40	Python
75	Lahdhirim/CV-human-pose-classifier-ViT-aws Human Pose Classifier using Vision Transformers (ViT) – end-to-end pipeline...	23	Experimental	5	Python
76	jman4162/PyTorch-Vision-Transformers-ViT Explore fine-tuning the Vision Transformer (ViT) model for object...	23	Experimental	7	Python
77	gianlucarloni/CoCoReco Code base for our paper "Connectivity-Inspired Network for Context-Aware...	22	Experimental	7	Python
78	Tejeshyewale/transfer_learning_in_Deeplearning This project demonstrates image classification using transfer learning with...	22	Experimental	—	Jupyter Notebook
79	Atharv279/Transfer-Learning Files containing projects related to Transfer Learning	22	Experimental	—	Jupyter Notebook
80	suous/RecNeXt RecConv: Efficient Recursive Convolutions for Multi-Frequency Representations	21	Experimental	19	Python
81	alantess/transformer Implementation of a modified vision transformer on the crypto market space	21	Experimental	14	Python
82	ViLab-UCSD/MemSAC_ECCV2022 PyTorch code for MemSAC. To appear in ECCV 2022.	21	Experimental	8	Jupyter Notebook
83	EthanBnntt/tinygrad-vit A minimalist implementation of the ViT (Vision Transformer) model, using tinygrad	20	Experimental	15	Python
84	RohanG9929/LoFTR-in-Tensorflow Code for our re-implementation of "LoFTR: Detector-Free Local Feature...	19	Experimental	8	Jupyter Notebook
85	PegHeads-Inc/PegHeads-Tutorial-4 TRANSFER LEARNING: TO CREATE A PRE-TRAINED MODEL	18	Experimental	6	Jupyter Notebook
86	OSU-MLB/ViT_PEFT_Vision [CVPR'25 (Highlight)] Lessons and Insights from a Unifying Study of...	18	Experimental	46	Jupyter Notebook
87	EmPasLab/ExMobileVIT ExMobileViT: Lightweight Classifier Extension for Mobile Vision Transformer	18	Experimental	5	Python
88	janaalbader28/Waste-Classification-ViT Exploring the use of Vision Transformers (ViT) for waste classification	17	Experimental	1	Jupyter Notebook
89	chinefed/convolutional-set-transformer Official implementation of the Convolutional Set Transformer (Chinello &...	16	Experimental	11	Jupyter Notebook
90	sanket-poojary-03/Fine-tuning-ViVit Python script to fine tune Open source Video Vision Transformer (ViVit)...	16	Experimental	14	Python
91	lizhh268/FSSUWNet [IJCNN 2025 Oral] Official implementation of paper: FSSUWNet: Mitigating the...	16	Experimental	3	—
92	zhouchenlin2096/Awesome-Transformer-for-Vision-Recognition A comprehensive paper list of Transformer & Attention for Vision Recognition...	15	Experimental	20	—
93	BobMcDear/vit-pytorch PyTorch implementation of the vision transformer	15	Experimental	17	Python
94	rentainhe/ViT.pytorch The Pytorch reimplementation of Vision Transformer	14	Experimental	10	Jupyter Notebook
95	EvgenyKashin/non-leaking-conv Implementation of Spectral Leakage and Rethinking the Kernel Size in CNNs in Pytorch	14	Experimental	14	Jupyter Notebook
96	AliKHaliliT/MobileViViT MobileViViT, a higher dimensional adaptation of MobileViT	14	Experimental	3	Python
97	VikramRangarajan/SIEDD A fast coordinate-based neural video encoder	14	Experimental	3	Python
98	zs1314/Fraesormer 【ICME2025 Oral】Offical Pytorch Code for "Fraesormer: Learning Adaptive...	14	Experimental	11	Python
99	kyegomez/LongVit A simplistic pytorch implementation of LongVit using my previous...	13	Experimental	7	Shell
100	dabane-ghassan/int-lab-book Foveated Spatial Transformers	13	Experimental	6	Jupyter Notebook
101	jiaowoguanren0615/DINOV2-Pytorch This is a warehouse for DinoV2-models, based pytorch framework.	13	Experimental	5	Python
102	eithannak29/NanoDiffVision NanoDiffVision explores Differential Attention as a natural evolution of...	13	Experimental	7	Python
103	nick8592/ViT-Classification-CIFAR10 This repository contains an implementation of the Vision Transformer (ViT)...	13	Experimental	6	Jupyter Notebook
104	MohammadRoodbari/Image-Classification image classification with fine tuning the BEiT vision transformer on CIFAR 10 dataset	13	Experimental	6	Jupyter Notebook
105	lucasjvds/ViT-for-Dark-Matter-Morphology Under the international Google Summer of Code program, the project...	12	Experimental	3	Jupyter Notebook
106	sntsemilio/Transfer-learning A machine learning project focused on transfer learning techniques using...	11	Experimental	—	Jupyter Notebook
107	iijumanaAhmed/Waste-Classification-ViT Exploring the use of Vision Transformers (ViT) for waste classification	11	Experimental	—	Jupyter Notebook
108	AriPathak/ViT-Berkley-CS198-HW4-Solution My pytorch implemented solution to the Fall 2020 UC Berkley CS198 ViT...	11	Experimental	—	Jupyter Notebook
109	OmarAlsaqa/GeoViG Implementation for GeoViG: Geometry-Aware Graph Reasoning for Mobile Vision...	11	Experimental	—	Python