DongmingShenDS/Mistral_From_Scratch
Mistral and Mixtral (MoE) from scratch
This project helps machine learning engineers and researchers understand and build large language models (LLMs) from the ground up. It provides step-by-step implementations of Mistral and Mixtral (Mixtral of Experts) architectures, including key components like RoPE, RMSNorm, and various attention mechanisms. Anyone looking to dive deep into the mechanics of modern LLMs will find this invaluable.
No commits in the last 6 months.
Use this if you are a machine learning engineer, researcher, or student who wants to learn the fundamental building blocks of advanced large language models by implementing them yourself.
Not ideal if you are an end-user simply looking to apply or fine-tune existing LLMs without needing to understand their internal mechanics.
Stars
9
Forks
—
Language
Python
License
—
Category
Last pushed
May 27, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/DongmingShenDS/Mistral_From_Scratch"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
mistralai/mistral-inference
Official inference library for Mistral models
dvmazur/mixtral-offloading
Run Mixtral-8x7B models in Colab or consumer desktops
open-compass/MixtralKit
A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI
vicuna-tools/vicuna-installation-guide
The "vicuna-installation-guide" provides step-by-step instructions for installing and...
pleisto/yuren-13b
Yuren 13B is an information synthesis large language model that has been continuously trained...