twang2218/vocab-coverage

Analyzes Chinese character recognition across language models through three complementary dimensions: character coverage rates against standard dictionaries (8,105–20,992 characters), tokenizer behavior patterns (WordPiece vs. byte-level BPE), and embedding space distributions via t-SNE visualization. The toolkit provides CLI commands (`charset`, `coverage`, `embedding`) to inspect vocabulary composition, measure character recognition depth (distinguishing single-token vs. fragmented representations), and assess whether token embeddings retain semantic structure or remain in random initialization. Comparative analysis spans BERT variants, ERNIE, multilingual models, LLaMA derivatives, and commercial APIs, revealing how tokenization strategies and training data affect Chinese language understanding.

236 stars and 42 monthly downloads. No commits in the last 6 months. Available on PyPI.