graykode/xlnet-Pytorch

Implements XLNet's permutation language modeling objective with Transformer-XL backbone, supporting configurable partial prediction masking, bidirectional data augmentation, and memory caching for long-context sequences. Uses SentencePiece/BERT tokenizers and exposes pretraining hyperparameters (sequence length, reuse length, permutation size, mask groups) as CLI arguments for reproducible experiments. Designed for PyTorch with minimal dependencies, enabling educational exploration of XLNet's two-stream self-attention and target-aware representations on custom datasets.

581 stars. No commits in the last 6 months.