MMMU-Benchmark/MMMU

Comprises 11.5K college-level multimodal questions spanning 6 disciplines and 183 subfields with 32 heterogeneous image types (charts, diagrams, chemical structures, etc.), designed to assess expert-level perception and reasoning. The evaluation framework includes both standard MMMU and MMMU-Pro variants, the latter employing stricter methodology—filtering text-only questions, augmenting distractors, and embedding questions as images to test simultaneous vision-and-reading capabilities. Supports systematic evaluation across development, validation, and test sets with local answer verification via JSON-based pipelines, enabling rigorous benchmarking of vision-language models without external submission infrastructure.

548 stars.