jasonacox/TinyLLM
Setup and run a local LLM and Chatbot using consumer grade hardware.
Supports multiple inference backends (Ollama, vLLM, llama-cpp-python) with OpenAI API compatibility, enabling flexible deployment across different hardware constraints. The chatbot layer adds RAG capabilities including URL summarization, news aggregation, stock/weather lookups, and vector database integration for knowledge retrieval. Architecture uses containerized services with persistent model caching and multi-session support via FastAPI frontend querying the inference backend.
319 stars.
Stars
319
Forks
37
Language
JavaScript
License
MIT
Category
Last pushed
Nov 23, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/jasonacox/TinyLLM"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
undreamai/LLMUnity
Create characters in Unity with LLMs!
Mintplex-Labs/anythingllm-docs
Documentation of AnythingLLM by Mintplex Labs Inc.
bloodworks-io/phlox
Open source, local first AI medical scribe for desktop and web.
mamei16/LLM_Web_search
An extension for oobabooga/text-generation-webui that enables the LLM to search the web
snexus/llm-search
Querying local documents, powered by LLM