sefineh-ai/Amharic-Tokenizer
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
When working with Amharic text for applications like machine translation or text analysis, you need to break down sentences into smaller, meaningful units. This tool takes raw Amharic text and outputs a list of 'tokens' (sub-word units) or numerical IDs for each token, which can then be used in various language models. It's designed for anyone building or researching natural language processing (NLP) systems for the Amharic language.
Use this if you need a fast and accurate way to prepare Amharic text for use in AI models, either by using a pre-trained model or training your own on custom Amharic data.
Not ideal if your project does not involve the Amharic language or if you need a full-fledged language model rather than just a tokenizer.
Stars
98
Forks
14
Language
Python
License
MIT
Category
Last pushed
Nov 17, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/sefineh-ai/Amharic-Tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
ml-rust/splintr
A high-performance tokenizer (BPE + SentencePiece) built with Rust with Python bindings, focused...
sanderland/script_tok
Code for the paper "BPE stays on SCRIPT"
ash-01xor/bpe.c
Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
U4RASD/r-bpe
R-BPE: Improving BPE-Tokenizers with Token Reuse