RulinShao/retrieval-scaling

Implements an efficient pipeline for building and serving retrieval-augmented language models at massive scale, supporting both dense retrievers (Contriever, SentenceTransformers) and sparse methods (BM25) with optional semantic chunking and memory-efficient passage loading via hashing. Provides distributed datastore construction and retrieval with optimized indexing (IVF-Flat, IVF-PQ) for sub-30ms API latency, plus integrated evaluation across perplexity and downstream tasks via adapted lm-evaluation-harness with published 1.4T-token and 140B-token datastores on Hugging Face.

224 stars.