THUDM/LongBench

Comprises 503 expert-curated multiple-choice questions with contexts ranging from 8k to 2M words across six task categories—single/multi-document QA, long in-context learning, dialogue understanding, code repos, and structured data. Evaluation harnesses vLLM for efficient inference serving with configurable tensor parallelism, supporting Chain-of-Thought reasoning and RAG-augmented testing modes. The benchmark exposes performance gaps between standard inference (50.1% best model accuracy) and reasoning-enhanced approaches like o1-preview (57.7%), demonstrating that scaling compute-intensive reasoning is critical for deep long-context understanding.

1,113 stars. No commits in the last 6 months.