LLaVA and llama-multimodal-vqa
LLaVA is a foundational vision-language instruction-tuning framework that llama-multimodal-vqa builds upon by adapting its techniques specifically for Llama 3 architecture and VQA tasks.
About LLaVA
haotian-liu/LLaVA
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
Combines a vision encoder (CLIP) with a lightweight projection layer to align image features with large language models, enabling end-to-end instruction tuning on image-text pairs. Supports efficient fine-tuning via LoRA, quantization (4/8-bit), and variable resolution inputs up to 4x higher pixel density in newer versions. Integrates with Hugging Face, llama.cpp, and AutoGen, with pre-trained checkpoints spanning multiple base models (LLaMA, Llama-2, Qwen, Llama-3).
About llama-multimodal-vqa
AdrianBZG/llama-multimodal-vqa
Multimodal Instruction Tuning for Llama 3
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work