hkchengrex/MMAudio

Leverages multimodal joint training across diverse audio-visual and audio-text datasets, enabling flexible video-to-audio or text-to-audio synthesis with a dedicated synchronization module (Synchformer) that temporally aligns generated audio to video frames. Built on PyTorch with CLIP-based visual encoding and operates efficiently in 16-bit precision (~6GB GPU memory), supporting variable input resolutions and frame rates through on-the-fly conversion.

2,115 stars.