Prompt Experimentation Platforms Prompt Engineering Tools

Tools for systematic A/B testing, comparison, and evaluation of LLM prompts across multiple models and variants. Includes statistical analysis, cost/performance measurement, and playground environments for prompt optimization. Does NOT include prompt templates, prompt collections, general LLM evaluation frameworks, or prompt management without experimentation features.

There are 29 prompt experimentation platforms tools tracked. The highest-rated is Supervertaler/Supervertaler-Workbench at 45/100 with 26 stars.

Get all 29 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=prompt-engineering&subcategory=prompt-experimentation-platforms&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	Supervertaler/Supervertaler-Workbench Open-source, AI-enhanced CAT tool with multi-LLM support, translation...	45	Emerging	26	Python
2	Mirascope/lilypad Open-source versioning, tracing, and annotation tooling.	41	Emerging	214	Python
3	crjaensch/PromptoLab A multi-platform app to serve as a prompts catalog, a LLM playground for...	37	Emerging	7	Python
4	parea-ai/parea-sdk-py Python SDK for experimenting, testing, evaluating & monitoring LLM-powered...	30	Emerging	82	Python
5	dbhavery/promptlab Prompt testing framework — pytest for LLM prompts. Define prompts as YAML,...	22	Experimental	—	Python
6	MukundaKatta/PromptLab Prompt experimentation workspace — A/B testing prompt variants with...	22	Experimental	—	Python
7	jeong-se-hun/autotune-skill Eval-first tuning skill for prompts, docs, skills, and code with guards,...	22	Experimental	—	Python
8	geeknees/sentinel_rb SentinelRb is an LLM-driven prompt inspector designed to automatically...	21	Experimental	2	Ruby
9	tmam-dev/tmam-python-sdk An open-source LLM engineering platform featuring observability, metrics,...	21	Experimental	—	Python
10	magifd2/log_analyzer A Python-based CLI tool for analyzing large log files (JSONL) with Large...	19	Experimental	—	Python
11	Personaz1/prompt-qa-lab Regression and evaluation toolkit for prompt and agent output quality	19	Experimental	—	Python
12	NeuroTinkerLab/synt-e-project A Python tool to translate natural language requests into efficient,...	19	Experimental	6	Python
13	vesper-astrena/promptlab Test and compare LLM prompts. Measure response time, tokens, and cost....	15	Experimental	1	Python
14	fernandoxx73/department-of-truth An experimental Python interface testing LLM constraint enforcement. It...	14	Experimental	—	Python
15	dakshjain-1616/promptfight Minimal prompt A/B testing: run two prompts 30 times, get winner + p-value +...	14	Experimental	—	Python
16	albipuliga/PromptLab Mange, test, and compare you prompts with different models.	14	Experimental	—	Vue
17	akashjindal423/Promptlab The open-source prompt engineering workbench. Analyse your LLM prompts...	14	Experimental	—	Python
18	rldyourmnd/local-llm-prompt-optimizer Offline prompt A/B testing, scoring & auto-tuning for local LLMs	14	Experimental	4	Python
19	prompt-foundry/python-sdk The prompt engineering, prompt management, and prompt evaluation tool for Python	13	Experimental	8	Python
20	artefactop/promptdev A prompt evaluation framework that provides comprehensive testing for AI...	13	Experimental	2	Python
21	martinklepsch/llm-web-ui A web UI for the `llm` command line tool	13	Experimental	6	TypeScript
22	mangobanaani/semantic-ui Minimal web interface for Large Language Models using Semantic Kernel	12	Experimental	1	Python
23	oruizramos/Blender-structured-knowledge-FAQ-retrieval PromptLab is a Python experimental framework for systematic prompt...	11	Experimental	—	Python
24	theishanpathak/prompt-tester Precision API analytics engine developed in Java 17 to track LLM usage...	11	Experimental	—	Java
25	EltonCN/toolpy Python module made to facilitate the creation of tools using LLMs.	11	Experimental	—	Python
26	joncoded/keywords keying in those words to understand them better (Next.js + Llama LLM + decap CMS)	11	Experimental	—	TypeScript
27	prompt-foundry/java-sdk The prompt engineering, prompt management, and prompt evaluation tool for Java.	10	Experimental	1	—
28	Shawn91/promtrix An intuitive GUI for evaluating and optimizing prompts and LLMs	10	Experimental	1	Python
29	prompt-foundry/ruby-sdk The prompt engineering, prompt management, and prompt evaluation tool for Ruby.	10	Experimental	1	—