Model Inference Serving ML Frameworks

Tools and frameworks for deploying, serving, and scaling machine learning models in production environments. Includes model servers, inference optimization, batching, and multi-model serving orchestration. Does NOT include model training frameworks, hyperparameter tuning, or general MLOps platforms.

There are 73 model inference serving frameworks tracked. 3 score above 70 (verified tier). The highest-rated is modelscope/modelscope at 90/100 with 8,784 stars and 3,316,702 monthly downloads. 3 of the top 10 are actively maintained.

Get all 73 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=ml-frameworks&subcategory=model-inference-serving&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Framework	Score	Tier	Stars	Language
1	modelscope/modelscope ModelScope: bring the notion of Model-as-a-Service to life.	90	Verified	8,784	Python
2	Lightning-AI/LitServe A minimal Python framework for building custom AI inference servers with...	78	Verified	3,812	Python
3	basetenlabs/truss The simplest way to serve AI/ML models in production	72	Verified	1,125	Python
4	tensorflow/serving A flexible, high-performance serving system for machine learning models	57	Established	6,349	C++
5	labmlai/labml 🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱	55	Established	2,300	Python
6	deepjavalibrary/djl-serving A universal scalable machine learning model deployment solution	55	Established	248	Java
7	OrderLab/TrainCheck An Observability Framework for AI Training	50	Established	66	Python
8	reacher-z/gpu-monitor Lightweight NVIDIA GPU monitor — alerts on Slack/Discord/Telegram/20...	46	Emerging	138	Python
9	iitzco/tfserve Serve TF models simple and easy as an HTTP API	45	Emerging	36	Python
10	awslabs/multi-model-server Multi Model Server is a tool for serving neural net models for inference	44	Emerging	1,025	Java
11	tobegit3hub/simple_tensorflow_serving Generic and easy-to-use serving service for machine learning models	44	Emerging	758	JavaScript
12	ShannonAI/service-streamer Boosting your Web Services of Deep Learning Applications.	42	Emerging	1,244	Python
13	VertaAI/modeldb Open Source ML Model Versioning, Metadata, and Experiment Management	42	Emerging	1,744	Java
14	polyaxon/sdks Polyaxon Clients & Langange SDKS	42	Emerging	14	Java
15	ZhigaMason/monitorch A plug-and-use python module to monitor neural network learning.	41	Emerging	6	Python
16	jrieke/traingenerator 🧙 A web app to generate template code for machine learning	41	Emerging	1,381	Python
17	ELS-RD/transformer-deploy Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗...	39	Emerging	1,688	Python
18	spotify/zoltar Common library for serving TensorFlow, XGBoost and scikit-learn models in production.	39	Emerging	142	Java
19	sustainable-computing-io/kepler-model-db Repository containing up-to-date models to be used by the kepler-model-server	38	Emerging	3	—
20	Angel-ML/serving A stand alone industrial serving system for angel.	38	Emerging	66	Java
21	Ifihan/blazerpc A lightweight, framework-agnostic gRPC library for serving machine learning...	36	Emerging	6	Python
22	zooniverse/bajor Azure Batch Job Runner - BaJoR	36	Emerging	2	Python
23	CODAIT/max-central-repo Central Repository of Model Asset Exchange project. This repository contains...	34	Emerging	218	—
24	feast-dev/feast-java-old Feast Java Components	34	Emerging	13	Java
25	alvarobartt/serving-pytorch-models Serving PyTorch models with TorchServe :fire:	34	Emerging	103	Jupyter Notebook
26	flipkart-incubator/Hunch Hunch allows users to turn arbitrary machine learning models built using...	33	Emerging	14	Python
27	rai-project/mlmodelscope MLModelScope is an open source, extensible, and customizable platform to...	33	Emerging	50	JavaScript
28	mKaloer/TFServingCache Distributed model cache for TF Serving	32	Emerging	25	Go
29	BeyonderXX/tensorflow-serving-tutorial A tutorial of building tensorflow serving service from scratch	31	Emerging	89	C++
30	ParagGhatage/ZeroML ZeroML is a visual-first, end-to-end machine learning platform that lets you...	30	Emerging	12	TypeScript
31	mme/vergeml Machine Learning Environment - alpha version	29	Experimental	339	Python
32	fuseml/fuseml-core FuseML APIs and core service. This repo include the FuseML client useful to...	28	Experimental	5	Go
33	kemingy/batching Dynamic Batching for Deep Learning Serving	28	Experimental	9	Go
34	ovh/serving-runtime Exposes a serialized machine learning model through a HTTP API.	28	Experimental	13	Java
35	Kenza-AI/kenza Open-Source Machine Learning Platform	27	Experimental	7	Go
36	huggingbench/huggingbench Find the optimal model serving solution for 🤗 Hugging Face models 🚀	26	Experimental	45	Python
37	bioinformatist/cml A Framework for Production-Ready Continuous Machine Learning	26	Experimental	6	Rust
38	alvarobartt/tensorflow-serving-streamlit TensorFlow Serving + Streamlit! :sparkles::framed_picture:	25	Experimental	24	Dockerfile
39	redis-applied-ai/redis-feast-gcp A demo of Redis Enterprise as the Online Feature Store deployed on GCP with...	25	Experimental	15	Jupyter Notebook
40	BBVA/pacarana A standalone ETL tool to generate advanced features for your Machine...	24	Experimental	4	Scala
41	tradingAI/runner Job runner for tbase experiments	23	Experimental	2	Go
42	kazuki-kanaya/obsern Lightweight CLI-based monitoring and notifications for long-running ML jobs....	23	Experimental	1	TypeScript
43	JohnJTK/crucible_train 🚀 Accelerate ML training on the BEAM with CrucibleTrain's unified...	22	Experimental	—	Elixir
44	puneethkotha/Falcon Production ML inference platform. Multi-worker · Nginx load balancing ·...	22	Experimental	—	Python
45	prabhuomkar/bitbeast Experiments with Model Training, Deployment & Monitoring	20	Experimental	40	Python
46	entrpn/serving-model-cards Collection of OSS models that are containerized into a serving container	20	Experimental	16	Python
47	datamass-io/ml-kraken Machine-Learning orchestration framework. Cloud-based models management environment.	20	Experimental	8	TypeScript
48	galafis/realtime-ml-serving-api High-performance ML model serving API built with Go and Python, featuring...	20	Experimental	1	Python
49	North-Shore-AI/crucible_train ML training orchestration for the Crucible ecosystem. Distributed training,...	19	Experimental	—	Elixir
50	HighviewOne/ml-model-registry ML Model Registry & Deployment Dashboard - AI Dev Tools Zoomcamp 2025	19	Experimental	—	TypeScript
51	North-Shore-AI/crucible_framework CrucibleFramework: A scientific platform for LLM reliability research on the BEAM	19	Experimental	—	Elixir
52	mcp-tool-shop-org/backprop CLI-first ML trainer with intelligent resource governance — timeboxed runs,...	19	Experimental	—	TypeScript
53	mmziyad/flink-ms Serving layer for large machine learning models on Apache Flink	18	Experimental	4	Java
54	North-Shore-AI/crucible_ensemble Multi-model ensemble voting strategies for LLM reliability	15	Experimental	—	Elixir
55	North-Shore-AI/crucible_adversary Adversarial testing and robustness evaluation for the Crucible framework	15	Experimental	—	Elixir
56	North-Shore-AI/crucible_deployment ML model deployment for the Crucible ecosystem. vLLM and Ollama integration,...	15	Experimental	—	Elixir
57	North-Shore-AI/crucible_feedback ML feedback loop management for the Crucible ecosystem. Quality monitoring,...	15	Experimental	—	Elixir
58	North-Shore-AI/crucible_model_registry ML model registry for the Crucible ecosystem. Artifact storage, model...	15	Experimental	—	Elixir
59	rsj-cs/distributed-pipeline-capstone Design and optimization of a scalable distributed data processing pipeline...	14	Experimental	—	—
60	galafis/distributed-model-inference-engine Distributed model inference engine with REST/gRPC serving, circuit breaker,...	14	Experimental	—	Python
61	man4ish/omnibioai-model-registry Production-grade model registry for the OmniBioAI ecosystem, providing...	14	Experimental	—	Python
62	kaniyeFelix/infinitetalk-deployment 🚀 Deploy InfiniteTalk and MultiTalk models effortlessly with automated...	14	Experimental	—	—
63	GAISSA-UPC/energy-ml-serving Energy consumption of ML inference with Runtime Engines	14	Experimental	3	Jupyter Notebook
64	aakashns/servefastai Serve FastAI models and get a web-based UI with a single line of code	12	Experimental	29	JavaScript
65	narphu/modelcache-operator Model caching in Kubernetes	11	Experimental	—	Go
66	arthurhzna/Golang_AI_Pipeline A scalable AI task queue and processing pipeline in Go, integrating Redis,...	11	Experimental	—	Go
67	gdroguski/MLServe Docker-based Machine Learning models serving	11	Experimental	2	Python
68	A-SHOJAEI/adaptive-inference-router-with-cascade-serving A research-grade adaptive inference routing system that learns to...	11	Experimental	—	Python
69	bluebell2505/qrucible Qrucible is a real‑world–aligned, decision‑support system for early‑stage...	11	Experimental	—	Jupyter Notebook
70	ameron-ai/model-serving-sidecar-service-example A simple Python example of a Model Service that can be fronted by the Model Sidecar	10	Experimental	1	Python
71	jonychoi/neuralverse Beyond the State of the Arts: Share more, Compare more, Edit Easy, Create...	10	Experimental	1	JavaScript
72	danielschulz/aiModelsAtScaleOnRestfulJeeSvcs Delivery Excellence, DevOps: Cloud-native Deployments of Data Science Models...	10	Experimental	1	Java
73	ameron-ai/model-serving-sidecar A lightweight adapter that handles all the cross-cutting concerns for model serving	10	Experimental	1	Java

Comparisons in this category

serving and simple_tensorflow_serving (57 vs 44)