eth-sri/lmql

Embeds LLM calls natively within Python syntax while applying declarative constraints via `where` clauses to guide model output—enabling control over token length, stopping phrases, and datatypes at the language level. Executes programs using advanced decoding strategies (beam search, best_k, argmax, sample) with optimizations like speculative execution and tree-based caching for faster inference. Supports OpenAI, Azure OpenAI, and Hugging Face Transformers models with async/parallel execution, plus integrations with LangChain and LlamaIndex.

4,161 stars. No commits in the last 6 months.