rahmanidashti/SyntheticTestCollections
[Official Codes] Synthetic Test Collections for Retrieval Evaluation (SIGIR 2024)
This project helps researchers and practitioners evaluate information retrieval (IR) systems by creating synthetic test collections. It takes existing documents and uses Large Language Models (LLMs) to generate new user queries and determine how relevant documents are to those queries, producing a complete test set. Information retrieval researchers, search engine developers, and anyone involved in assessing IR system performance would use this.
No commits in the last 6 months.
Use this if you need to evaluate an information retrieval system but face challenges in acquiring diverse user queries and costly manual relevance judgments.
Not ideal if you already have access to extensive, human-generated test collections and relevance judgments, or if you specifically require evaluating systems only with real user data.
Stars
11
Forks
—
Language
—
License
—
Category
Last pushed
Jul 19, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/rahmanidashti/SyntheticTestCollections"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
google/langfun
OO for LLMs
tanaos/artifex
Small Language Model Inference, Fine-Tuning and Observability. No GPU, no labeled data needed.
preligens-lab/textnoisr
Adding random noise to a text dataset, and controlling very accurately the quality of the result
vulnerability-lookup/VulnTrain
A tool to generate datasets and models based on vulnerabilities descriptions from @Vulnerability-Lookup.
masakhane-io/masakhane-mt
Machine Translation for Africa