marklysze/LlamaIndex-RAG-WSL-CUDA

Demonstrates RAG with quantized GGUF models running locally via llama-cpp-python, optimized for Nvidia CUDA acceleration in WSL environments, and includes Word document ingestion with source attribution. Built on LlamaIndex Core with configurable prompt templates for different model architectures, enabling semantic search across documents with retrieval-augmented generation capabilities. Provides production-ready notebooks comparing outputs across multiple model sizes (2B–34B parameters) to assess quality-versus-resource tradeoffs.

132 stars. No commits in the last 6 months.