smeetrs/deep_avsr

Implements three independent models (audio-only, video-only, audio-visual) using a TM-CTC architecture trained on LRS2 for speech-to-text transcription, with audio-visual fusion demonstrating significant robustness gains (6.8% WER vs. 54% for audio-only in noisy conditions). Training employs curriculum learning with progressive vocabulary expansion and transfer learning between modalities, supported by greedy and beam search decoders with language model integration. Targets PyTorch with CUDA 10.0, includes preprocessing and inference pipelines compatible with LRS2 dataset specifications.

243 stars. No commits in the last 6 months.