kolenaIO/autoarena

Implements comparative evaluation using Elo-ranked pairwise comparisons from multiple LLM judges (OpenAI, Anthropic, Cohere, or custom) rather than isolated metrics, grounded in research showing head-to-head judgments are more reliable. Employs a "Panel of LLMs" (PoLL) approach using diverse smaller models simultaneously for better accuracy at lower cost than single frontier models. Provides a local web UI with SQLite storage, CSV-based prompt/response ingestion, and support for custom judges connecting to internal services.

108 stars. No commits in the last 6 months.