asahi417/lm-vocab-trimmer

Vocabulary Trimming (VT) is a model compression technique, which reduces a multilingual LM vocabulary to a target language by deleting irrelevant tokens from its vocabulary. This repository contains a python-library vocabtrimmer, that remove irrelevant tokens from a multilingual LM vocabulary for the target language.

33
/ 100
Emerging

No commits in the last 6 months.

Stale 6m No Package No Dependents
Maintenance 0 / 25
Adoption 8 / 25
Maturity 16 / 25
Community 9 / 25

How are scores calculated?

Stars

63

Forks

5

Language

Python

License

MIT

Category

llm-compression-optimization

Last pushed

Oct 25, 2024

Commits (30d)

0

GitHub
Llm Compression Optimization · 29 models

Get this data via API

curl "https://pt-edge.onrender.com/api/v1/quality/transformers/asahi417/lm-vocab-trimmer"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

Higher-rated alternatives

ModelTC/LightCompress

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs,...

67

p-e-w/heretic

Fully automatic censorship removal for language models

65

YerbaPage/LongCodeZip

LongCodeZip: Compress Long Context for Code Language Models [ASE2025]

54

Orion-zhen/abliteration

Make abliterated models with transformers, easy and fast

54

zyushun/Adam-mini

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

48

Explore Transformer Models

All categories Trending Transformer directory Insights