lyuchenyang/Macaw-LLM

Combines CLIP for image/video encoding, Whisper for audio encoding, and LLaMA as the language backbone, with a lightweight attention-based alignment layer that bridges multi-modal embeddings to LLM token space. Introduces one-stage instruction fine-tuning and a 119K-example multi-modal instruction dataset (69K image-based, 50K video-based) generated from MS COCO, Charades, and AVSD captions using GPT-3.5-Turbo. Supports processing of all four modalities in unified inference, with minimal additional parameters compared to the base LLM.

1,593 stars. No commits in the last 6 months.