qiangsiwei/bert_distill

Implements knowledge distillation from BERT into lightweight student models (TextCNN, BiLSTM/GRU) using both Keras and PyTorch, following the approach in "Distilling Task-Specific Knowledge from BERT into Simple Neural Networks." The framework uses a 1:8:1 labeled-to-unlabeled-to-test data split and supports data augmentation techniques (masking, n-gram sampling) to improve student model performance on sentiment classification tasks, achieving ~87-88% accuracy compared to BERT's ~90-91% on the clothing dataset.

314 stars. No commits in the last 6 months.