tatsu-lab/alpaca_eval

Implements length-controlled win-rate scoring to mitigate output length bias, achieving 0.98 Spearman correlation with ChatBot Arena while costing under $10 and completing in under 3 minutes. Uses LLM-based pairwise comparisons (GPT-4 by default) against a reference model, validated against 20K human annotations with built-in caching and output randomization. Provides a toolkit for constructing custom evaluators with batching and multi-annotator support, plus curated evaluation datasets and leaderboards for benchmarking instruction-following models.

1,957 stars. No commits in the last 6 months.