fresh-stack/freshstack
This repository helps you evaluate your models on the FreshStack benchmark!
Supports both dense embedding models (via BEIR) and multi-vector retrieval systems (via PyLate/ColBERT), enabling comprehensive evaluation of diverse retrieval architectures. The framework automatically constructs benchmarks from real Stack Overflow queries and GitHub repository code, evaluated using nugget-based metrics (α-NDCG, coverage, recall) that capture partial relevance across multiple valid answers. Datasets and evaluation scripts are provided for five technical topics with standardized BEIR format compatibility.
Available on PyPI.
Stars
33
Forks
3
Language
Python
License
Apache-2.0
Category
Last pushed
Dec 09, 2025
Monthly downloads
43
Commits (30d)
0
Dependencies
3
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/embeddings/fresh-stack/freshstack"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Related tools
embeddings-benchmark/mteb
MTEB: Massive Text Embedding Benchmark
yannvgn/laserembeddings
LASER multilingual sentence embeddings as a pip package
harmonydata/harmony
The Harmony Python library: a research tool for psychologists to harmonise data and...
embeddings-benchmark/results
Data for the MTEB leaderboard
Hironsan/awesome-embedding-models
A curated list of awesome embedding models tutorials, projects and communities.