VKCOM/YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
ArchivedImplements Byte Pair Encoding with O(N) complexity using multithreaded C++ backend and space-as-boundary tokenization (preserving word boundaries via "▁" meta-symbol). Provides Python bindings and CLI tools supporting BPE-dropout regularization and reversible encoding/decoding. Outperforms Hugging Face tokenizers, fastBPE, and SentencePiece by up to 60× on training and inference through efficient parallel processing.
975 stars. No commits in the last 6 months.
Stars
975
Forks
109
Language
C++
License
MIT
Category
Last pushed
Mar 29, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/VKCOM/YouTokenToMe"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
soaxelbrooke/python-bpe
Byte Pair Encoding for Python!
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer