brightmart/bert_language_understanding

Implements masked language model pre-training with TextCNN as an alternative backbone to Transformers, demonstrating that pre-training objectives are architecture-agnostic. Built in TensorFlow, it supports both single-label and multi-label text classification with configurable model sizes (big/small/tiny) via d_model and attention head parameters. Pre-training on raw unlabeled data yields significant performance gains—achieving 0.75 F1 after 7 fine-tuning epochs versus 0.44 after 35 epochs without pre-training on mid-sized datasets.

967 stars. No commits in the last 6 months.