meta-llama/synthetic-data-kit
Tool for generating high quality Synthetic datasets
Supports multi-format document ingestion (PDF, DOCX, HTML, YouTube transcripts) and generates structured fine-tuning datasets through a modular 4-stage pipeline: ingest → create (QA pairs, Chain-of-Thought reasoning, or summaries) → curate (using Llama-as-judge quality filtering) → save-as (converts to Alpaca, OpenAI, or HuggingFace formats). Uses Lance vector storage by default and integrates with vLLM or external LLM APIs for generation, with full YAML-based configuration overrides.
1,524 stars. Available on PyPI.
Stars
1,524
Forks
215
Language
Python
License
MIT
Category
Last pushed
Oct 28, 2025
Commits (30d)
0
Dependencies
15
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/meta-llama/synthetic-data-kit"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related frameworks
Diyago/Tabular-data-generation
We well know GANs for success in the realistic image generation. However, they can be applied in...
tdspora/syngen
Open-source version of the TDspora synthetic data generation algorithm.
always-further/deepfabric
Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline
Data-Centric-AI-Community/ydata-synthetic
Synthetic data generators for tabular and time-series data
wiseodd/generative-models
Collection of generative models, e.g. GAN, VAE in Pytorch and Tensorflow.