microsoft/aici

Implements constrained decoding through lightweight WebAssembly modules that execute on the inference engine's CPU during token generation, enabling token-by-token control over LLM output. Controllers written in Rust, C++, Python, or JavaScript maintain state and implement diverse strategies—from programmatic decoding to multi-agent coordination—while the sandbox architecture ensures security without filesystem or network access. Integrates with llama.cpp, HuggingFace Transformers, and rLLM, designed as a portable layer for higher-level control libraries like Guidance and LMQL.

2,064 stars. No commits in the last 6 months.