jdh-algo/JoyVASA

Decouples static 3D facial representations from dynamic motion sequences using a diffusion transformer trained on audio features (wav2vec2 or HuBERT), enabling identity-independent motion generation that extends to animal face animation. The two-stage pipeline extracts 3D facial appearance via LivePortrait, generates motion sequences from speech in a sliding-window fashion, and renders final video through keypoint-based warping and a learned generator. Supports multilingual audio input and is compatible with both portrait and animal image animation through optional MultiScaleDeformableAttention components.

856 stars.