JindongGu/Awesome-Prompting-on-Vision-Language-Model

Organized curated papers across three distinct VLM categories—multimodal-to-text generation (Flamingo), image-text matching (CLIP), and text-to-image generation (Stable Diffusion)—with taxonomies distinguishing hard prompts (task instructions, in-context learning, retrieval-based, chain-of-thought) from soft prompts (prompt tuning, prefix token tuning). The collection emphasizes methods that preserve base model weights while adapting VLMs through fusion module architectures (encoder-decoder vs. decoder-only) and input augmentation strategies. Includes linked implementations, publication venues, and technical notes for each paper to support systematic research into prompt engineering across multimodal foundation models.

509 stars. No commits in the last 6 months.