b4rtaz/distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Implements tensor parallelism across a root-worker architecture where the root node loads and distributes model slices over Ethernet to worker nodes, with synchronized state updates using configurable float precision (q80/f32). Supports CPU inference on ARM and x86_64 with AVX2 optimization, plus experimental Vulkan GPU acceleration, enabling models like Llama 3.3 70B and Qwen 3 MoE across commodity hardware with quantized weights (Q40).
2,856 stars.
Stars
2,856
Forks
215
Language
C++
License
MIT
Category
Last pushed
Feb 10, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/b4rtaz/distributed-llama"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related models
Blaizzy/mlx-vlm
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac...
armbues/SiLLM
SiLLM simplifies the process of training and running Large Language Models (LLMs) on Apple...
microsoft/batch-inference
Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
armbues/SiLLM-examples
Examples for using the SiLLM framework for training and running Large Language Models (LLMs) on...
kolinko/effort
An implementation of bucketMul LLM inference