Cloud-CV/EvalAI
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
Supports remote evaluation on distributed worker clusters and Docker-based code submissions for agent evaluation in isolated environments. Built on Django, Node.js, PostgreSQL, and Docker with map-reduce backends for parallel dataset processing and warm-loaded worker nodes that pre-import challenge code and datasets to minimize evaluation latency. Includes a CLI tool and allows custom evaluation protocols with arbitrary phases, dataset splits, and public/private leaderboards compatible with any programming language.
2,013 stars and 538 monthly downloads. Actively maintained with 1 commit in the last 30 days. Available on PyPI.
Stars
2,013
Forks
989
Language
Python
License
—
Category
Last pushed
Mar 12, 2026
Monthly downloads
538
Commits (30d)
1
Dependencies
11
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/Cloud-CV/EvalAI"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related frameworks
fireindark707/Python-Schema-Matching
A python tool using XGboost and sentence-transformers to perform schema matching task on tables.
graphbookai/graphbook
Visual AI development framework for training and inference of ML models, scaling pipelines, and...
visual-layer/fastdup
fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and...
josh-ashkinaze/plurals
Plurals: A System for Guiding LLMs Via Simulated Social Ensembles
github/CodeSearchNet
Datasets, tools, and benchmarks for representation learning of code.