soaxelbrooke/python-bpe
Byte Pair Encoding for Python!
232 stars and 995 monthly downloads. No commits in the last 6 months. Available on PyPI.
Stars
232
Forks
39
Language
Python
License
MIT
Category
Last pushed
Sep 16, 2022
Monthly downloads
995
Commits (30d)
0
Dependencies
6
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/soaxelbrooke/python-bpe"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
taishi-i/toiro
A tool for comparing tokenizers