thom-heinrich/chonkify
Extractive document compression for RAG and agent pipelines. +69% vs LLMLingua, +175% vs LLMLingua2 on information recovery. Compiled wheels, try it out.
Builds document units scored through 768-dimensional embeddings and selects the highest-ranked segments to stay within token budgets while maximizing factual recovery—critical for quantitative research and reasoning traces where exact facts outweigh fluent paraphrasing. Supports multiple embedding backends including Azure OpenAI, OpenAI-compatible APIs, and fully offline local SentenceTransformers, with a CLI and Python API for RAG pipelines and agent memory systems. Ships as compiled extension modules for performance-sensitive workloads across Linux, Windows, and macOS platforms.
Stars
5
Forks
2
Language
Python
License
—
Category
Last pushed
Mar 26, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/thom-heinrich/chonkify"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Higher-rated alternatives
chonkie-inc/chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust...
speedyk-005/chunklet-py
One library to split them all: Sentence, Code, Docs. Chunk smarter, not harder — built for LLMs,...
andreshere00/Splitter_MR
Chunk your data into markdown text blocks for your LLM applications
chonkie-inc/chonkiejs
🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library
jchunk-io/jchunk
JChunk is a lightweight and flexible library designed to provide multiple strategies for text...