microsoft/unilm

Built on foundation architectures like DeepNet (1,000+ layer Transformers) and Magneto (general-purpose multimodal modeling), the project implements unified pre-training across diverse modalities including vision (BEiT, DiT), speech (WavLM, VALL-E), and document understanding (LayoutLM series). It emphasizes training stability and efficiency through techniques like sparse Mixture-of-Experts (X-MoE) and length extrapolation, while supporting 100+ languages via models like InfoXLM and DeltaLM for cross-lingual transfer and machine translation.

22,042 stars.