oneKn8/bengali-tokenizer-eval

Tokenizer choice can make Bengali 5-9x more expensive. Paper, evaluation framework, Unicode failure analysis, and 13 released tokenizers.

/ 100

Experimental

No Package No Dependents

Maintenance 13 / 25

Adoption 0 / 25

Maturity 9 / 25

Community 0 / 25

Stars

—

Forks

—

Language

BibTeX Style

License

MIT

Last pushed

Apr 02, 2026

Commits (30d)

Get this data via API

curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/oneKn8/bengali-tokenizer-eval"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

Higher-rated alternatives

openai/openai-python

The official Python library for the OpenAI API

pydantic/pydantic

Data validation using Python type hints

campfirein/byterover-cli

ByteRover CLI (brv) - The portable memory layer for autonomous coding agents (formerly Cipher)

mistralai/client-python

Python client library for Mistral AI platform

cohere-ai/cohere-python

Python Library for Accessing the Cohere API