airaria/TextBrewer

Supports multiple distillation techniques including attention-matrix matching, hidden state MSE, and neuron selectivity transfer, with flexible intermediate layer matching and multi-teacher configurations. Built on a modular framework that enables custom loss functions and distillers without requiring model architecture modifications, compatible with Transformer-based architectures across diverse NLP tasks. Integrates with the Hugging Face Transformers library and supports distributed training via DistributedDataParallel and mixed-precision training with Apex.

1,697 stars. No commits in the last 6 months.