Text Tokenization Libraries NLP Tools

There are 9 text tokenization libraries tools tracked. 2 score above 50 (established tier). The highest-rated is guillaume-be/rust-tokenizers at 61/100 with 336 stars and 7,405 monthly downloads.

Get all 9 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=nlp&subcategory=text-tokenization-libraries&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	guillaume-be/rust-tokenizers Rust-tokenizer offers high-performance tokenizers for modern language...	61	Established	336	Rust
2	sugarme/tokenizer NLP tokenizers written in Go language	54	Established	316	Go
3	elixir-nx/tokenizers Elixir bindings for 🤗 Tokenizers	46	Emerging	100	Elixir
4	openscilab/tocount ToCount: Lightweight Token Estimator	42	Emerging	21	Python
5	reinfer/blingfire-rs Rust wrapper for the BlingFire tokenization library	37	Emerging	15	Rust
6	frothywater/kanade-tokenizer Kanade is a single-layer disentangled speech tokenizer that extracts compact...	37	Emerging	85	Python
7	euskadi31/go-tokenizer A Text Tokenizer library for Golang	32	Emerging	11	Go
8	howl-anderson/PaddleTokenizer 使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 \| A DNN Chinese Tokenizer by Using PaddlePaddle	25	Experimental	15	JavaScript
9	Mecanik/Modern-Text-Tokenizer Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and...	16	Experimental	12	C++

Comparisons in this category

tokenizer and go-tokenizer (54 vs 32)