lechmazur/confabulations

Evaluates RAG systems by measuring both confabulation rates (false answers to unanswerable questions) and non-response rates across 201 human-verified adversarial questions and 2,612 answerable questions, revealing performance tradeoffs that simple accuracy metrics miss. Combines confabulation and non-response scoring into weighted rankings, exposing models that avoid hallucinations by refusing to answer entirely. Tests at temperature 0 across major LLM providers without relying on model-based evaluation, which the author found introduces significant bias.

243 stars. No commits in the last 6 months.