Rust LLM Infrastructure LLM Tools

Low-level Rust libraries and tools for building, running, and managing LLMs locally—including model merging, inference engines, tokenization, and architecture implementations. Does NOT include application frameworks, API clients, or higher-level orchestration platforms.

There are 104 rust llm infrastructure tools tracked. 1 score above 50 (established tier). The highest-rated is trymirai/uzu at 56/100 with 1,492 stars. 1 of the top 10 are actively maintained.

Get all 104 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=llm-tools&subcategory=rust-llm-infrastructure&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	trymirai/uzu A high-performance inference engine for AI models	56	Established	1,492	Rust
2	lipish/llm-connector LLM Connector - A unified interface for connecting to various Large Language...	49	Emerging	13	Rust
3	justrach/bhumi ⚡ Bhumi – The fastest AI inference client for Python, built with Rust for...	47	Emerging	64	Python
4	rustformers/llm [Unmaintained, see README] An ecosystem of Rust libraries for working with...	44	Emerging	6,152	Rust
5	keyvank/femtoGPT Pure Rust implementation of a minimal Generative Pretrained Transformer	42	Emerging	935	Rust
6	kreuzberg-dev/liter-llm Universal LLM API client — 142+ providers, 11 native language bindings,...	41	Emerging	4	Rust
7	mplekh/rust-microgpt Port of Andrej Karpathy's python microGPT to Rust	38	Emerging	36	Rust
8	ShelbyJenkins/llm_client The Easiest Rust Interface for Local LLMs and an Interface for Deterministic...	38	Emerging	245	Rust
9	luckenco/rsai Predictable development for unpredictable models. Let the compiler handle the chaos.	37	Emerging	9	Rust
10	EggerMarc/tools-rs Serialize your functions with tools-rs!	36	Emerging	6	Rust
11	InfraWhisperer/llmtop htop for your LLM inference cluster	36	Emerging	6	Go
12	haasonsaas/uranium High-security storage vault for Large Language Model (LLM) weights with...	36	Emerging	2	Rust
13	npuichigo/openai_trtllm OpenAI compatible API for TensorRT LLM triton backend	36	Emerging	219	Rust
14	visualstudioblyat/yule Run AI models locally Prove what ran	35	Emerging	2	Rust
15	antirez/gguf-tools GGUF implementation in C as a library and a tools CLI program	34	Emerging	311	C
16	microsoft/aici AICI: Prompts as (Wasm) Programs	34	Emerging	2,064	Rust
17	darkautism/llmserver-rs A Rust-based, OpenAI-style API server for large language models (LLMs)	34	Emerging	19	Rust
18	brontoguana/ktop Terminal system resource monitor for hybrid LLM workloads	33	Emerging	64	Rust
19	reinterpretcat/qwen3-rs An educational Rust project for exporting and running inference on Qwen3 LLM family	33	Emerging	40	Rust
20	Michael-A-Kuykendall/schoolmarm Production-grade GBNF grammar-constrained decoding for LLMs. Zero...	32	Emerging	1	Rust
21	fabriziopfannl/llm-autobatch Turn single LLM calls into fast micro-batches. Rust core, Python API.	31	Emerging	4	Rust
22	FerrisMind/inspector-gguf A powerful GGUF file inspection tool with a graphical and command-line interface	30	Emerging	1	Rust
23	Mattbusel/llm_affector An async Rust library for LLM-based content analysis, providing...	30	Emerging	5	Rust
24	rosarp/llm-lsp Language Server Protocol for accessing Large Language Models	30	Emerging	42	Rust
25	Lallapallooza/gpt.rs Rust LLM playground: build, train, generate on pluggable backends	30	Emerging	15	Rust
26	yigitkonur/cli-batch-requester 10K+ req/s batch API client for LLM endpoints — Rust, async, load-balanced	30	Emerging	19	Rust
27	tmetsch/rusty_llm Rust based AI LLM inference service	29	Experimental	6	Rust
28	GammaTauAI/opentau Using Large Language Models for Repo-wide Type Prediction	27	Experimental	114	Rust
29	uky007/FerrugoCC Rust-based reverse optimization (code obfuscation) C Compiler	27	Experimental	2	Rust
30	Mattbusel/llm-sync CRDT and vector clock primitives for distributed LLM agent state synchronization	27	Experimental	3	Rust
31	paiml/apr-cookbook Examples of .apr format models	26	Experimental	5	Rust
32	Mattbusel/llm-wasm LLM inference primitives for WebAssembly — cache, retry, routing, guards,...	26	Experimental	2	Rust
33	Mattbusel/llm-diff Output diffing and versioning for LLM outputs — semantic diff, version...	26	Experimental	2	Rust
34	aprxi/talu Talu is a single-binary, local-first LLM runtime with a Zig core and...	26	Experimental	7	Zig
35	codito/arey Simple large language model playground app	26	Experimental	6	Rust
36	netdur/hugind vLLM for poor GPUs	26	Experimental	6	Rust
37	rodmarkun/flyllm A Rust library for unifying LLM backends as an abstraction layer with load...	25	Experimental	28	Rust
38	jaggederest/locque Locque, a dependently-typed LLM first programming language	25	Experimental	3	Haskell
39	okayasl/normy Ultra-fast, zero-copy text normalization for Rust NLP pipelines & tokenizers	25	Experimental	3	Rust
40	Jack17432/positivity A Rust crate that provides a generic method to determine non-negativity for...	24	Experimental	2	Rust
41	hyperpolymath/patallm-gallery Gallery of LLM patterns and implementations	24	Experimental	2	Rust
42	antoineMoPa/rust-text-experiments Tiny LLM in rust / candle	24	Experimental	2	Rust
43	lspecian/crabinfer Safe, fast, memory-aware on-device LLM inference SDK for iOS — built in Rust...	24	Experimental	2	Rust
44	chenhunghan/mlx-training-rs A CLI in Rust to generate synthetic data for MLX friendly training	23	Experimental	25	Rust
45	richardanaya/epistemology A simple and clear way of hosting llama.cpp as a private HTTP API using Rust	23	Experimental	26	Rust
46	yybit/pllm Portable LLM - A rust library for LLM inference	23	Experimental	11	Rust
47	jondot/awesome-rust-llm 🦀 A curated list of Rust tools, libraries, and frameworks for working with...	23	Experimental	543	—
48	HelgeSverre/sema A Lisp with first-class LLM primitives, implemented in Rust	23	Experimental	1	Rust
49	qora-protocol/QORA-LLM-3B Pure Rust inference engine for the SmolLM3-3B language model. No Python...	23	Experimental	1	Rust
50	usemarbles/langmail Email preprocessing for LLMs. Fast, typed, Rust-powered.	23	Experimental	1	Rust
51	TomOst-Sec/BlueOS GPU-first LLM inference runtime in Rust + CUDA. Tiered virtual VRAM,...	23	Experimental	1	Rust
52	sizzlecar/ferrum-infer-rs Rust-native LLM inference engine. Single binary, no Python. Chat locally or...	23	Experimental	1	Rust
53	greysquirr3l/heretic-rs Abliterate LLMs in pure Rust — zero Python, single static binary, runs on Colab	22	Experimental	—	Rust
54	GoWtEm/llm-model-selector A high-performance Rust utility that analyzes your system hardware to...	22	Experimental	4	Rust
55	wassemgtk/llm-training-rust llm training rust	22	Experimental	1	Rust
56	qwrtgvdsdf/ternary-tools 🔍 Explore and validate GGUF files effortlessly with ternary-tools, a...	22	Experimental	—	Rust
57	Ranjitbarnala0/rai CPU-native LLM inference engine — hand-written SIMD kernels, 4-bit...	22	Experimental	—	Rust
58	proj-airi/candle-examples 🦀 Rust powered LLM, Whisper, Embedding inference, backed by 🤗 candle from HuggingFace	22	Experimental	30	Rust
59	cukas/KERNlang The language LLMs think in. Write one .kern file, ship 7 targets. 70% fewer tokens	22	Experimental	—	TypeScript
60	petlukk/Cougar Fast, dependency-free LLM engine in Rust with custom SIMD kernels	22	Experimental	—	Rust
61	defai-digital/ax-engine Mac-native Rust inference engine for running larger local GGUF models with...	22	Experimental	—	Rust
62	PCfVW/candle-mi Mechanistic interpretability for language models in Rust, built on candle	22	Experimental	—	Rust
63	TheRadDani/VectorPrime VectorPrime takes a model file and your hardware, then finds the fastest way...	22	Experimental	—	Rust
64	SundryAPI/sundry Sundry is an intelligent context provider API designed specifically for...	22	Experimental	9	Rust
65	nkypy/candle-rwkv RWKV models and examples powered by candle.	21	Experimental	24	Rust
66	pwh-pwh/couplet_gen use rust to generate couplet	21	Experimental	8	Rust
67	t81dev/ternary-tools file(1) of the ternary age — balanced-ternary-aware GGUF inspector and...	20	Experimental	1	Rust
68	coconut-os/coconutOS Rust microkernel for GPU-isolated AI inference	20	Experimental	1	Rust
69	Defilan/gguf-parser A Rust library and CLI for parsing GGUF model file headers — extract...	20	Experimental	1	Rust
70	ahoylabs/gguf.js A Javascript library (with Typescript types) to parse metadata of GGML based...	20	Experimental	51	TypeScript
71	neuron-nexus-agregator/nn-yandex-foundation Unified library for working with Yandex Foundation Models. Provides a simple...	19	Experimental	7	Rust
72	yarenty/modelmux ModelMux is a high-performance Rust proxy server that seamlessly converts...	19	Experimental	—	Rust
73	menezis-ai/LDSI White-box LLM stability benchmark using Kolmogorov complexity, Shannon...	19	Experimental	—	Rust
74	tzervas/axolotl-rs YAML-driven configurable fine-tuning toolkit for LLMs in Rust	19	Experimental	—	Rust
75	chongliujia/fermi-infer The Rust-native inference engine for Small Language Models (SLMs), Run...	19	Experimental	—	Rust
76	AspadaX/secretary Robustly create/extract structural data with LLMs	18	Experimental	5	Rust
77	matthewhaynesonline/phile Single file llm, but in _rust_. phi + file = phile.	18	Experimental	3	Rust
78	santino-research/spell A Programming Language Designed for Large Language Models	18	Experimental	3	Rust
79	blueheron786/cpu_llm A lightweight CPU-friendly neural language model from scratch, with hybrid...	17	Experimental	2	Rust
80	srijitiyer/alloy A fast Rust CLI for LLM model merging, diffing, and conversion. 10 merge...	17	Experimental	4	Rust
81	cjroth/neuroscope Real-time "x-ray vision" into LLMs' minds	17	Experimental	3	Rust
82	msk/lumine A high-level Rust interface for language models powered by the Candle ML...	16	Experimental	1	Rust
83	JuliaMerz/pantry Actor based multi-llm registry + runner.	15	Experimental	20	Rust
84	abdulrahmanashraf5594/comprehensive-rust 🦀 Explore Comprehensive Rust, a multi-day course that teaches Rust from...	15	Experimental	1	Dart
85	reinterpretcat/zero-depend-pub An educational Rust workspace featuring zero-dependency crates built using...	15	Experimental	—	Rust
86	samkeen/llm-bridge Rust SDK for interacting with various Large Language Model (LLM) APIs	14	Experimental	4	Rust
87	StepfenShawn/ferris-grad Pytorch-like autograd engine in Rust.	14	Experimental	3	Rust
88	lipish/llm-providers A unified source of truth for LLM providers, models, pricing, and...	14	Experimental	—	Rust
89	Plarturer/llm-distributed-inference High-performance distributed inference engine for LLMs using Rust and CUDA.	14	Experimental	—	Rust
90	sanggi-wjg/LLML LLML — Language for Large Model Logic. A programming language optimized for...	14	Experimental	—	Rust
91	ltouati/tiny-llm A tiny llm writen using rust candle	14	Experimental	—	Rust
92	eren23/synapse Modular LLM inference engine in Rust + Zig SIMD kernels. Runs on desktop...	14	Experimental	—	Rust
93	rhi-zone/sketchpad Deep learning inference in pure Rust using Burn. Image generation (SD, SDXL,...	12	Experimental	1	Rust
94	llmprogram/llmprogram-rs llmprogram is a Rust crate that provides a structured and powerful way to...	11	Experimental	—	Rust
95	gicrisf/microgpt-candle-rs Rust implementation of Karpathy's Microgpt	11	Experimental	—	Rust
96	kn0sys/adamo Rust LLM proof-of-concept	11	Experimental	—	Rust
97	magic003/llama2-rs Inference Llama 2 in Rust	11	Experimental	—	Rust
98	text-yoga/ask WIP browser-based LLM question/answering for the web	11	Experimental	2	Rust
99	tauseefk/streamformers Wrap Rustformers' LLM inference in a stream.	11	Experimental	2	Rust
100	kmolerov/llm-temp-scale llm-temp-scale is a multiplatform library for normalizing and converting a...	11	Experimental	—	Rust
101	zTgx/transformer-rust Transformer With Rust & Candle	11	Experimental	—	Rust
102	AshtonVaughan/prismllm Any model. Any hardware. Any size. — Hardware-agnostic LLM inference with...	11	Experimental	—	Rust
103	ramendrasingla/ml_algorithms_in_rust Creating Machine Learning and Deep Learning Algorithms in Rust	10	Experimental	1	Rust
104	mrcsparker/guanaco Run local LLMs in Ruby	10	Experimental	1	Rust

Comparisons in this category

llm-connector and liter-llm (49 vs 41)