pjlab-sys4nlp/llama-moe

Converts dense LLaMA FFN layers into sparse mixture-of-experts through neuron partitioning (random, clustering, co-activation graph, or gradient-based) and top-K gating, maintaining only 3.0–3.5B activated parameters. Supports multiple gating strategies (TopK Noisy Gate, Switch Gating) and optimizes training via FlashAttention-v2 integration with dynamic batch sampling weights from Sheared LLaMA. Integrates with Hugging Face Transformers ecosystem and provides comprehensive training monitoring (gate load/importance, load balancing metrics, throughput visualization).

1,002 stars. No commits in the last 6 months.