Mecanik/Tiny-BPE-Trainer

Lightweight, header-only Byte Pair Encoding (BPE) trainer in modern C++17. Produces HuggingFace-compatible vocabularies for transformers and integrates with Modern Text Tokenizer.

14
/ 100
Experimental

No commits in the last 6 months.

Stale 6m No Package No Dependents
Maintenance 2 / 25
Adoption 3 / 25
Maturity 9 / 25
Community 0 / 25

How are scores calculated?

Stars

4

Forks

—

Language

C++

License

MIT

Category

tokenizer-libraries

Last pushed

Aug 08, 2025

Commits (30d)

0

GitHub
Tokenizer Libraries · 20 models

Get this data via API

curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Mecanik/Tiny-BPE-Trainer"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

Featured in

We Audited crewAI's AI Dependencies: Here's What the Data Says

Higher-rated alternatives

huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

90

megagonlabs/ginza-transformers

Use custom tokenizers in spacy-transformers

48

Kaleidophon/token2index

A lightweight but powerful library to build token indices for NLP tasks, compatible with major...

48

NVIDIA/Cosmos-Tokenizer

A suite of image and video neural tokenizers

42

Hugging-Face-Supporter/tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

38

Explore Transformer Models

All categories Trending Transformer directory Insights