bytedance/LatentSync
Taming Stable Diffusion for Lip Sync!
Audio-conditioned latent diffusion operating directly in Stable Diffusion's latent space, using Whisper-extracted audio embeddings injected via U-Net cross-attention layers. Trains with TREPA, LPIPS, and SyncNet losses in pixel space while working on compressed latents, avoiding intermediate motion representations. Supports multi-resolution training (256×256 to 512×512) with configurable efficiency modes, ranging from 20–55 GB VRAM depending on stage and resolution.
5,506 stars. No commits in the last 6 months.
Stars
5,506
Forks
899
Language
Python
License
Apache-2.0
Category
Last pushed
Jun 20, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/diffusion/bytedance/LatentSync"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related models
PrunaAI/pruna
Pruna is a model optimization framework built for developers, enabling you to deliver faster,...
haoheliu/AudioLDM-training-finetuning
AudioLDM training, finetuning, evaluation and inference.
Text-to-Audio/Make-An-Audio
PyTorch Implementation of Make-An-Audio (ICML'23) with a Text-to-Audio Generative Model
Aratako/Irodori-TTS
A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control
sayakpaul/diffusers-torchao
End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8...