google-research/electra

Employs a replaced token detection objective using a generator-discriminator setup inspired by GANs, where a smaller transformer generates plausible token replacements and the main model learns to identify fakes—enabling efficient pre-training on single GPUs while achieving competitive downstream performance. Supports fine-tuning on classification (GLUE), QA (SQuAD), and sequence tagging tasks, with an alternative "Electric" variant that frames training as energy-based cloze modeling and enables pseudo-likelihood scoring for text re-ranking. Built on TensorFlow 1.15 with tfrecord-based pre-training pipelines and includes pre-trained checkpoints (Small/Base/Large) alongside code for continued pre-training from released weights.

2,371 stars. No commits in the last 6 months.