ML Robustness Frameworks

Tools and frameworks for testing, evaluating, and improving the robustness of ML models against corruptions, adversarial perturbations, domain shifts, and distribution changes. Does NOT include general model evaluation, fairness/bias mitigation, or privacy-preserving machine learning outside robustness contexts.

There are 34 ml robustness frameworks tracked. 1 score above 50 (established tier). The highest-rated is namkoong-lab/dro at 56/100 with 157 stars and 133 monthly downloads.

Get all 34 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=ml-frameworks&subcategory=ml-robustness-frameworks&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Framework	Score	Tier	Stars	Language
1	namkoong-lab/dro A package of distributionally robust optimization (DRO) methods. Implemented...	56	Established	157	Jupyter Notebook
2	THUDM/grb Graph Robustness Benchmark: A scalable, unified, modular, and reproducible...	49	Emerging	99	Python
3	neu-autonomy/nfl_veripy Formal Verification of Neural Feedback Loops (NFLs)	48	Emerging	83	Python
4	iutzeler/skwdro Distributionally robust machine learning with Pytorch and Scikit-learn wrappers	45	Emerging	19	Python
5	MinghuiChen43/awesome-trustworthy-deep-learning A curated list of trustworthy deep learning papers. Daily updating...	45	Emerging	382	—
6	ADA-research/VERONA A lightweight Python package for setting up robustness experiments and to...	42	Emerging	49	Python
7	hendrycks/robustness Corruption and Perturbation Robustness (ICLR 2019)	41	Emerging	1,139	Python
8	microsoft/robustdg Toolkit for building machine learning models that generalize to unseen...	37	Emerging	175	Python
9	alibaba/easyrobust EasyRobust: an Easy-to-use library for state-of-the-art Robust Computer...	35	Emerging	338	Jupyter Notebook
10	Iyengar-Lab/E2E-DRO End-to-end distributionally robust optimization	33	Emerging	38	Python
11	jiachens/ModelNet40-C Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against...	33	Emerging	216	Python
12	RyanLucas3/HR_Neural_Networks Certified robustness of deep neural networks	31	Emerging	19	Python
13	RuntianZ/doro Distributional and Outlier Robust Optimization (ICML 2021)	30	Emerging	28	Python
14	MLI-lab/Robustness-CS Measuring the robustness of compressive sensing methods (including...	29	Experimental	32	Jupyter Notebook
15	BBVA/mercury-robust mercury-robust is a framework to perform robust testing on ML models and...	29	Experimental	20	Jupyter Notebook
16	ShoumikSaha/DRSM DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified...	29	Experimental	14	Python
17	CausalML/doubly-robust-dropel Off-Policy Evaluation and Learning that is both Doubly Robust and...	28	Experimental	10	Jupyter Notebook
18	val-iisc/GD-UAP Generalized Data-free Universal Adversarial Perturbations	28	Experimental	73	Python
19	sarthaxxxxx/AVROBUSTBENCH Benchmarking robustness of audio-visual recognition models at test-time	27	Experimental	10	Jupyter Notebook
20	jh-jeong/smoothmix Code for the paper "SmoothMix: Training Confidence-calibrated Smoothed...	26	Experimental	21	Roff
21	pmichel31415/P-DRO Code for the papers "Modeling the Second Player in Distributionally Robust...	25	Experimental	29	Python
22	ByungKwanLee/Super-Fast-Adversarial-Training Official PyTorch Implementation Code for Developing Super Fast Adversarial...	25	Experimental	33	Python
23	Trustworthy-ML-Lab/corrupting_neuron_explanations [ICCV 23] Evaluating robustness of neuron explanation methods	24	Experimental	4	Jupyter Notebook
24	or4k2l/robust-vision Production-ready framework for training robust computer vision models....	23	Experimental	1	Python
25	AI-secure/VeriGauge A united toolbox for running major robustness verification approaches for...	20	Experimental	90	C
26	alejandrods/Analysis-of-the-robustness-of-NMF-algorithms Analysis of the robustness of non-negative matrix factorization (NMF)...	18	Experimental	11	Jupyter Notebook
27	SebChw/Actually-Robust-Training Actually Robust Training - Tool Inspired by Andrej Karpathy "Recipe for...	17	Experimental	43	Python
28	nmndeep/revisiting-at [NeurIPS 2023] Code for the paper "Revisiting Adversarial Training for...	16	Experimental	39	Python
29	katelyn98/CorruptionRobustness We investigated corruption robustness across different architectures...	16	Experimental	16	Jupyter Notebook
30	dedeswim/vits-robustness-torch Code for the paper "A Light Recipe to Train Robust Vision Transformers" [SaTML 2023]	16	Experimental	54	Jupyter Notebook
31	sghosh-04/cnn-generalization-dataset-shift independent research on CNN robustness under dataset shift — EfficientNet,...	14	Experimental	—	Jupyter Notebook
32	im-ethz/pub-gdu4dg Gated Domain Units (GDU) aim to make your deep learning models robust...	13	Experimental	7	Jupyter Notebook
33	MK-Wireless/coded-neural-networks Structured redundancy for neural networks using coding-theoretic principles.	11	Experimental	—	—
34	Yangyi-Chen/PaperList-Trustworthy-Applications Mostly recording papers about models' trustworthy applications. Intending to...	11	Experimental	21	—

Comparisons in this category

dro and E2E-DRO (56 vs 33) dro and skwdro (56 vs 45)