ChenghaoMou/text-dedup

Provides four deduplication algorithms—MinHash with LSH for near-duplicates, SimHash for semantic similarity, Bloom filters for exact matches, and suffix arrays for substring deduplication—all configured via TOML. Uses a modular architecture supporting multiple input formats (parquet, local files) with configurable thresholds, hash parameters, and merge strategies. Includes comprehensive benchmarks on public datasets (CORE, NEWS-COPY) demonstrating competitive performance against embedding-based and hybrid approaches.

746 stars and 695 monthly downloads. Actively maintained with 2 commits in the last 30 days. Available on PyPI.