scientist-labs/tokenkit

Fast, Rust-backed word-level tokenization for Ruby. Unlike subword tokenizers (BPE, WordPiece) designed for LLMs, TokenKit provides linguistic tokenization for search engines, text mining, and NLP pipelinesā€”preserving domain-specific patterns like gene names, measurements, and technical terms while handling Unicode correctly.

22
/ 100
Experimental
No Package No Dependents
Maintenance 10 / 25
Adoption 3 / 25
Maturity 9 / 25
Community 0 / 25

How are scores calculated?

Stars

3

Forks

—

Language

Ruby

License

MIT

Category

tokenization-algorithms

Last pushed

Mar 06, 2026

Commits (30d)

0

GitHub
Tokenization Algorithms · 57 tools

Get this data via API

curl "https://pt-edge.onrender.com/api/v1/quality/nlp/scientist-labs/tokenkit"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

Higher-rated alternatives

google/sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

84

soaxelbrooke/python-bpe

Byte Pair Encoding for Python!

61

OpenNMT/Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support

59

Systemcluster/kitoken

Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...

58

daac-tools/vibrato

šŸŽ¤ vibrato: Viterbi-based accelerated tokenizer

57

Explore NLP Tools

All categories Trending NLP directory Insights