x-transformers and simple-hierarchical-transformer
X-transformers is a general-purpose transformer library that simple-hierarchical-transformer builds upon as an experimental architecture variant, making them complements rather than competitors.
About x-transformers
lucidrains/x-transformers
A concise but complete full-attention transformer with a set of promising experimental features from various papers
Supports encoder-decoder, decoder-only (GPT), and encoder-only (BERT) architectures alongside vision transformers for image classification and multimodal tasks like image captioning and vision-language modeling. Implements experimental attention mechanisms including Flash Attention for memory-efficient training, persistent memory augmentation, and memory tokens, while offering fine-grained control over dropout strategies including stochastic depth and layer-wise dropout. Built as a PyTorch library with modular components (`TransformerWrapper`, `Encoder`, `Decoder`, `ViTransformerWrapper`) enabling flexible composition for tasks ranging from language modeling to vision-language understanding.
About simple-hierarchical-transformer
lucidrains/simple-hierarchical-transformer
Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work