majiayu000/litellm-rs

Offers intelligent routing with load balancing, failover, and cost optimization across providers, plus built-in caching, rate limiting, and auth for enterprise deployments. Supports multiple modalities (chat, embeddings, images, audio) and can run as either a lightweight library for API aggregation or a full gateway server with SQLite/Redis storage. Achieves 10,000+ requests/second with <10ms routing overhead through async Tokio architecture and modular feature flags that let users pay only for what they need.