rohitrango/objects-that-sound

Trains AVENet on AudioSet to learn joint audio-visual embeddings through unsupervised correspondence learning, enabling cross-modal retrieval without labeled data. The architecture encodes video frames and spectrograms into a shared embedding space, allowing queries like "find images matching this sound" or vice versa via Euclidean distance on learned representations. Implements the full training pipeline on a 46k-video subset with handling for real-world dataset challenges like temporal misalignment, class imbalance, and multi-label noise.

No commits in the last 6 months.