tokenizer and go-tokenizer

These are competitors offering overlapping functionality—both provide standalone tokenization libraries for Go without clear differentiation in scope or features, so a user would select one based on maturity (sugarme/tokenizer's higher star count suggests broader adoption) rather than using them together.

tokenizer
54
Established
go-tokenizer
32
Emerging
Maintenance 6/25
Adoption 10/25
Maturity 16/25
Community 22/25
Maintenance 6/25
Adoption 5/25
Maturity 9/25
Community 12/25
Stars: 316
Forks: 61
Downloads:
Commits (30d): 0
Language: Go
License: Apache-2.0
Stars: 11
Forks: 2
Downloads:
Commits (30d): 0
Language: Go
License: MIT
No Package No Dependents
No Package No Dependents

About tokenizer

sugarme/tokenizer

NLP tokenizers written in Go language

About go-tokenizer

euskadi31/go-tokenizer

A Text Tokenizer library for Golang

Scores updated daily from GitHub, PyPI, and npm data. How scores work