LLaVA and ViP-LLaVA
ViP-LLaVA builds upon LLaVA's architecture by extending its visual instruction tuning approach to handle arbitrary visual prompts (like spatial markers and annotations) rather than just image-text pairs, making them complementary advances in the same multimodal instruction-tuning lineage.
About LLaVA
haotian-liu/LLaVA
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
Combines a vision encoder (CLIP) with a lightweight projection layer to align image features with large language models, enabling end-to-end instruction tuning on image-text pairs. Supports efficient fine-tuning via LoRA, quantization (4/8-bit), and variable resolution inputs up to 4x higher pixel density in newer versions. Integrates with Hugging Face, llama.cpp, and AutoGen, with pre-trained checkpoints spanning multiple base models (LLaMA, Llama-2, Qwen, Llama-3).
About ViP-LLaVA
WisconsinAIVision/ViP-LLaVA
[CVPR2024] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work