xlang-ai/OSWorld

Provides a unified benchmarking environment supporting multiple VM backends (VMware, VirtualBox, Docker with KVM, AWS) for evaluating multimodal agents on realistic desktop tasks, with parallelized evaluation capabilities reducing runtime to under an hour. The framework captures screen observations and executes agent actions through desktop automation APIs, enabling systematic assessment of end-to-end task completion across diverse applications requiring real OS interactions. Includes 747+ curated tasks spanning web browsing, file management, and productivity software, plus baseline results for vision-language models and agentic frameworks.

2,664 stars. Actively maintained with 31 commits in the last 30 days.