kyegomez/Sophia
Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.
Implements a second-order optimization algorithm using inexpensive stochastic Hessian diagonal estimation (via Hutchinson or Gauss-Newton-Bartlett methods) with gradient clipping to precondition parameter updates. Supports both SophiaG and DecoupledSophia variants that integrate directly into PyTorch training loops without architectural changes, requiring minimal hyperparameter tuning beyond learning rate adjustment (typically half of AdamW's value) and rho clipping bounds.
381 stars. No commits in the last 6 months.
Stars
381
Forks
26
Language
Python
License
Apache-2.0
Category
Last pushed
Jun 04, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/kyegomez/Sophia"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
LLM-Tuning-Safety/LLMs-Finetuning-Safety
We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 adversarially...
uthmandevsec/Self-Distillation
🤖 Enable continual learning by reproducing the On-Policy Self-Distillation algorithm for robust...
appier-research/robust-llm-finetunes
Accepted to NeurIPS 2025
jmcentire/apprentice
Train cheap models on expensive ones. Automatically. With receipts.
phonism/LLMNotes
LLM 学习笔记：Transformer 架构、强化学习 (RLHF/DPO/PPO)、分布式训练、推理优化。含完整数学推导与Slides。