TonyLianLong/LLM-groundedDiffusion

Uses LLMs to parse natural language prompts into spatial layouts (bounding boxes with captions), which then guide diffusion models via cross-attention control and optional GLIGEN adapters for improved compositional generation. Integrated into HuggingFace Diffusers (v0.24.0+) and supports both proprietary APIs (GPT-3.5/4) and self-hosted open-source LLMs (Mixtral, LLaMA 2), with built-in caching, SDXL refinement, and multiple layout-to-image methods (GLIGEN, MultiDiffusion, BoxDiff).

481 stars. No commits in the last 6 months.