Whisper Fine-Tuning Voice AI Tools

Tools and frameworks for fine-tuning Whisper models on custom datasets, including language-specific adaptation, accent conditioning, and model distillation. Does NOT include pre-built Whisper applications, deployment wrappers, or inference optimization without training components.

There are 27 whisper fine-tuning tools tracked. 1 score above 50 (established tier). The highest-rated is YuanGongND/whisper-at at 57/100 with 412 stars and 1,194 monthly downloads.

Get all 27 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=voice-ai&subcategory=whisper-fine-tuning&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	YuanGongND/whisper-at Code and Pretrained Models for Interspeech 2023 Paper "Whisper-AT:...	57	Established	412	Python
2	huggingface/distil-whisper Distilled variant of Whisper for speech recognition. 6x faster, 50% smaller,...	45	Emerging	4,056	Python
3	adi-gov-tw/Taiwan-Tongues-ASR-CE Taiwan Tongues ASR CE 是一個開源語音辨識（Automatic Speech Recognition,...	40	Emerging	29	Python
4	KevKibe/African-Whisper 🚀 Framework for seamless fine-tuning of Whisper model on a multi-lingual...	37	Emerging	37	Python
5	sandy1990418/ChineseTaiwaneseWhisper This repository focuses on leveraging OpenAI's Whisper model for speech...	34	Emerging	70	Python
6	ga642381/Taiwanese-Whisper fine-tune Whipser model for Taiwanese speech recognition	32	Emerging	37	Python
7	phineas-pta/fine-tune-whisper-vi jupyter notebooks to fine tune whisper models on Vietnamese using Colab...	32	Emerging	19	Jupyter Notebook
8	EtienneAb3d/WhisperHallu Experimental code: sound file preprocessing to optimize Whisper...	32	Emerging	348	Python
9	LianjiaTech/bella-whisper bella-whisper是一系列基于OpenAI...	29	Experimental	3	Python
10	huuquyet/PhoWhisper-next Demo using PhoWhisper models of VinAI built with Transformers.js + Next.js	28	Experimental	7	TypeScript
11	fengredrum/finetune-whisper-lora Fine-Tune Whisper with Transformers and PEFT	27	Experimental	58	Python
12	tonywu71/distilling-and-forgetting-in-large-pre-trained-models Code for my dissertation on "Distilling and Forgetting in Large Pre-Trained...	24	Experimental	2	Jupyter Notebook
13	HiMeditator/wfts-chinese-tool 使用中文游玩《群星低语》游戏。Playing the game "Whisper from the Stars" in Chinese.	23	Experimental	8	Python
14	sovse/base_rus_whisper_stt Fine tuning of the base model from OpenAI Whisper in Russian language on the...	23	Experimental	40	Jupyter Notebook
15	naver/multilingual-distilwhisper This repository contains all the code necessary for running the multilingual...	21	Experimental	33	Python
16	petitwhito/Speech_to_text_project Complete Speech-to-Text pipeline: from-scratch architectures (MLP, CNN, RNN,...	16	Experimental	1	Python
17	sonhm3029/Realtime-Vietnamese-ASR-React-Native-and-Whisper This project implement end to end realtime vietnamese speech recognition...	16	Experimental	4	JavaScript
18	my-north-ai/semantic_audio_filtering Synthetic data augmentation technique via LLM for Automatic Speech...	16	Experimental	10	Python
19	runze123/cantonese-asr-evaluation This project presents a systematic evaluation of two state-of-the-art...	15	Experimental	1	Python
20	HKAB/vietnamese-rnnt-tutorial A tutorial on how to train RNN-T from scratch with Whisper encoder	14	Experimental	12	Python
21	10809104/taigi-speech-to-text 台語語音轉文字訓練資料集，資料來源：教育部《臺灣閩南語常用詞辭典》。	12	Experimental	1	Python
22	mavleo96/whisper-accent Conditioning via Adaptive Layer Norm for accented speech recognition	12	Experimental	1	Python
23	navalnica/whisper-finetuning-be Finetuning Whisper ASR model for Belarusian language	12	Experimental	17	Jupyter Notebook
24	innerNULL/simpler-distil-whisper Simpler Distil-Whisper	12	Experimental	3	Python
25	thomas-ferraz/Whisper-Robustness Fine-tuning SOTA Speech Foundation Model (Whisper) for Speech Transcription	12	Experimental	3	Python
26	Bilel-Eljaamii/Whisper-Arabic-Poetry-Performance Benchmarking OpenAI Whisper models (tiny→turbo) for classical Arabic poetry...	11	Experimental	2	Python
27	bivex/whisper-large-v3-turbo Whisper Large V3 Turbo - fast speech-to-text model implementation with...	11	Experimental	—	Python

Comparisons in this category

ChineseTaiwaneseWhisper and Taiwanese-Whisper (34 vs 32)