pranav271103/Ultra-Tokenizer
This project implements a state-of-the-art tokenizer from scratch in Python, designed to be more efficient and flexible than existing tokenizers like BERT's.
No commits in the last 6 months. Available on PyPI.
Stars
2
Forks
—
Language
Python
License
Apache-2.0
Category
Last pushed
Aug 16, 2025
Monthly downloads
23
Commits (30d)
0
Dependencies
3
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/pranav271103/Ultra-Tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
soaxelbrooke/python-bpe
Byte Pair Encoding for Python!
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer