nomic-ai/contrastors

Leverages Flash Attention custom kernels and GradCache to enable memory-efficient training at scale, supporting multi-GPU distributed training with large batch sizes. Integrates with Hugging Face model hubs (BERT, Pythia, ViT via timm) and implements advanced techniques including Matryoshka Representation Learning, CLIP/LiT-style objectives, and MLM pretraining. Includes streaming data ingestion from cloud storage (Cloudflare R2) with byte-offset indexing for datasets exceeding available memory, demonstrated through production embeddings like Nomic Embed.

777 stars. No commits in the last 6 months.