google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
Implements both byte-pair-encoding (BPE) and unigram language model algorithms with subword regularization techniques to improve model robustness. Operates directly on raw Unicode text without requiring language-specific preprocessing, and provides end-to-end vocabulary-to-ID mapping with NFKC normalization. Available as self-contained C++ and Python libraries that achieve ~50k sentences/sec throughput while maintaining consistent tokenization across deployments.
11,697 stars and 33,078,873 monthly downloads. Used by 194 other packages. Actively maintained with 2 commits in the last 30 days. Available on PyPI.
Stars
11,697
Forks
1,333
Language
C++
License
Apache-2.0
Category
Last pushed
Mar 01, 2026
Monthly downloads
33,078,873
Commits (30d)
2
Reverse dependents
194
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/google/sentencepiece"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related tools
soaxelbrooke/python-bpe
Byte Pair Encoding for Python!
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
taishi-i/toiro
A tool for comparing tokenizers