princeton-nlp/SimPO

Replaces the reference model dependency in DPO with a simplified reward formulation based on implicit rewards from margin-based losses, eliminating computational overhead while maintaining performance. Integrates with HuggingFace Transformers and the TRL trainer framework, with support for both on-policy and offline preference data across Llama, Mistral, and Gemma model families. Demonstrates state-of-the-art results on AlpacaEval 2, MT-Bench, and Arena-Hard benchmarks through careful hyperparameter tuning of learning rate, beta (reward scaling), and gamma (target margin).

946 stars. No commits in the last 6 months.