kuleshov-group/mdlm

Combines discrete diffusion with a substitution-based parameterization that reduces to masked language modeling losses, enabling competitive perplexity on LM1B and OpenWebText. Implements multiple samplers including a novel cached variant achieving 3-4× speedup over prior diffusion LMs, plus semi-autoregressive generation at 25-30× faster decoding than comparable models. Supports DiT and Mamba architectures, integrates with Hugging Face Hub, and provides complete training pipelines via PyTorch Lightning with SLURM job scripts.

657 stars. No commits in the last 6 months.