cactus-compute/cactus

Provides OpenAI-compatible APIs for multimodal inference (chat, vision, speech, embeddings, RAG) with zero-copy memory mapping that reduces RAM usage 10x, leveraging a custom PyTorch-based computation graph and ARM SIMD kernels optimized for different chipsets (Apple, Snapdragon, Exynos). Supports automatic cloud fallback for requests exceeding device capabilities and NPU-accelerated prefill for models ranging from 270M to 8B parameters across iOS, Android, macOS, and Linux via C, C++, Python, Swift, Kotlin, Flutter, Rust, and JavaScript SDKs.

4,430 stars. Actively maintained with 54 commits in the last 30 days.