x-transformers and TransformerX
These are competitors: x-transformers is a mature, production-ready transformer implementation with experimental features, while TransformerX is an earlier-stage research library offering modular building blocks for the same purpose of implementing transformer architectures.
About x-transformers
lucidrains/x-transformers
A concise but complete full-attention transformer with a set of promising experimental features from various papers
Supports encoder-decoder, decoder-only (GPT), and encoder-only (BERT) architectures alongside vision transformers for image classification and multimodal tasks like image captioning and vision-language modeling. Implements experimental attention mechanisms including Flash Attention for memory-efficient training, persistent memory augmentation, and memory tokens, while offering fine-grained control over dropout strategies including stochastic depth and layer-wise dropout. Built as a PyTorch library with modular components (`TransformerWrapper`, `Encoder`, `Decoder`, `ViTransformerWrapper`) enabling flexible composition for tasks ranging from language modeling to vision-language understanding.
About TransformerX
tensorops/TransformerX
Flexible Python library providing building blocks (layers) for reproducible Transformers research (Tensorflow ✅, Pytorch 🔜, and Jax 🔜)
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work