guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
336 stars and 7,405 monthly downloads.
Stars
336
Forks
33
Language
Rust
License
Apache-2.0
Category
Last pushed
Jan 22, 2026
Monthly downloads
7,405
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/guillaume-be/rust-tokenizers"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
sugarme/tokenizer
NLP tokenizers written in Go language
elixir-nx/tokenizers
Elixir bindings for 🤗 Tokenizers
openscilab/tocount
ToCount: Lightweight Token Estimator
reinfer/blingfire-rs
Rust wrapper for the BlingFire tokenization library
frothywater/kanade-tokenizer
Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for...