--- title: Visual In-Context Learning (V-ICL) url: https://www.emergentmind.com/topics/visual-in-context-learning-v-icl type: topic --- # Visual In-Context Learning (V-ICL) Visual In-Context Learning (V-ICL) is the extension of the in-context learning (ICL) paradigm, originally discovered in large language models, to the domain of vision and multimodal tasks. In V-ICL, a model adapts to new visual or vision-language tasks during inference by conditioning on a small set of demonstration examples, without updating its parameters. Crucially, both the construction of the context sequence (prompts/demonstrations) and the architectural strategies for encoding and leveraging visual information are central to V-ICL’s empirical performance and generalization capabilities. V-ICL spans a continuum from pure visual models operating on images or spatial representations, to vision-language models handling arbitrary multi-modal sequences involving images, captions, and complex reasoning instructions. ## 1. Foundational Principles and Unified V-ICL Formulation The canonical V-ICL setup defines a task whereby a frozen model is presented with $k$ demonstration pairs—each a combination of a visual input (e.g., image, image patch, segmentation mask) and a corresponding output (e.g., mask, caption, label)—followed by a query input. The objective is to autoregressively predict the correct output for the query, solely by composing the provided demonstrations in context. A general formalization is: - Given $k$ demonstrations $\{(x^{(i)}_\text{vis}, y^{(i)})\}_{i=1}^k$ and a query $x_\text{vis}^{k+1}$, the model produces $y^{k+1}$ such that $$ P_\theta(y^{k+1} | x_\text{vis}^{k+1}, \{x^{(i)}_\text{vis}, y^{(i)}\}_{i=1}^k) $$ Standard approaches proceed by quantizing images and (if present) text into a unified discrete token space via VQGAN (for images, masks) and BPE tokenization (for text), then embedding these as interleaved token sequences [2312.02520]. The unified V-ICL transformer architecture, as exemplified by recent works, operates on interleaved sequences of embedded tokens—spanning both modalities—using sparse attention mechanisms and mixture-of-experts (MoE) feed-forward blocks, with all outputs generated by the same decoder stack [2312.02520]. In context, the model is evaluated by simply prepending the demonstration pairs to the query in the constructed sequence and generating the required output, with no adaptation or re-weighting. ## 2. Demonstration Selection, Prompt Construction, and Retrieval Strategies The performance of V-ICL is highly sensitive to how in-context demonstrations are selected, ordered, and composed within the prompt. Early approaches relied on k-nearest neighbor (kNN) retrieval using pretrained image-embedding spaces (e.g., CLIP, ViT, DINOv2), often by computing the feature similarity between the query and each pool element and selecting the most similar [2304.04748, 2312.06592]. However, similarity-based retrieval can lead to overly redundant contexts, particularly in regression or attribute-diverse tasks. Recent research reframes demonstration selection as an optimization or sequential decision-making problem. The Learning to Select Demonstrations (LSD) framework formulates demonstration selection as a finite-horizon Markov Decision Process, training a dueling Deep Q-Network to compose demonstration sets that optimize downstream MLLM performance via marginal error reduction [2603.26775]. LSD avoids the pitfall of selecting demonstrations too similar to the query and instead actively maximizes the diversity and label-space coverage of the context. Notably, LSD yields substantial improvements over kNN on objective regression tasks, whereas kNN remains near optimal for subjective, preference-based settings. Prompt fusion—how demonstrations and queries are spatially and positionally composed—matters as much as prompt selection. Pixel-level fusion and grid-based positional arrangements, along with ensembling over multiple permutations, further improve robustness and accuracy [2304.04748]. Flexible memory approaches, such as those adapted from Video Object Segmentation (VOS), allow variable-sized in-context supports and avoid grid resolution bottlenecks [2312.06592]. ## 3. Architectures, Unification, and Generative Modeling Unified V-ICL architectures are based on a decoder-only transformer, with a single token vocabulary and embedding for both visual and textual modalities [2312.02520]. Visual inputs (images, patches, masks) are first passed through an encoder (typically convolutional or vision transformer), quantized via learned codebooks (e.g., VQGAN), and then embedded identically to text tokens. This enables the model to process and relate image→image, image→text, and potentially other modalities in a single pipeline. Key architectural features include: - **Sparse Self-Attention:** Block-local and strided sparsity patterns are used for scalability [2312.02520]. - **Mixture-of-Experts (MoE):** Every other transformer block uses an MoE FFN, which is critical for reducing task interference (e.g., preventing segmentation and captioning signals from degrading each other’s performance) [2312.02520]. - **Autoregressive Next-Token Loss:** All supervision occurs via next-token prediction over every token in the interleaved sequence (including query output tokens), optionally with an auxiliary load-balancing loss for MoE blocks. For particular tasks, additional design considerations apply. In instance-level object localization, attention regularizers are employed to focus transformer attention from the query image to support bounding-box tokens, and reinforcement learning objectives directly reward high IoU [2605.31145]. In the context of segmentation, prompt-supporting learners based on flexible, memory-based architectures—where image/mask pairs are stored as memory without grid concatenation—offer improved generalization to unseen classes and better support set scaling [2312.06592]. Other works leverage learnable prompt perturbations (PEFT) to shift the latent representation of the prompt closer to the target task distribution, providing consistent gains, especially in cases of poor dataset coverage [2504.18158]. ## 4. Empirical Benchmarks, Task Coverage, and Evaluation Regimes V-ICL has been systematically evaluated across a broad spectrum of tasks: semantic segmentation, object detection, image captioning, visual regression, classification, and vision-language reasoning. Benchmarks such as VL-ICL Bench present challenges from perception and recognition to multimodal operator induction and long-context interleaving [2403.13164]. Key findings include: - On class-aware segmentation, unified V-ICL achieves mean IoU (MIoU) of 58.04 on MS-COCO (256×256) with a 309M parameter model, surpassing task-specific baselines under matching resolution [2312.02520]. - In joint captioning tasks, V-ICL’s unified pipeline yields competitive BLEU4, METEOR, CIDEr, and mAP scores compared to specialized models [2312.02520]. - Quantitative ablations demonstrate that prompt interleaving, MoE structure, and multi-task sampling are all essential for stable, high-quality multimodal ICL [2312.02520]. - On multimodal reasoning, state-of-the-art VLLMs (e.g., GPT-4V) exhibit genuine in-context learning but only for a limited number of examples, as higher shot counts can lead to context length overloads and performance degradation [2403.13164]. ## 5. Strengths, Limitations, and Open Challenges ### Strengths - **Unified Multimodal Pipeline:** A single backbone handles image→image, image→text, and potentially arbitrary quantized modalities [2312.02520]. - **True Zero-Update Adaptation:** New tasks can be solved by demonstrating a few context pairs without any fine-tuning or gradient steps. - **Scalability Across Tasks:** With appropriate architectural choices (MoE, interleaving, memory-based prompting), the model can co-train or adapt across multiple tasks. - **Demonstration Optimization:** Reinforcement learning-based demonstration selection markedly improves objective regression tasks over baseline retrieval [2603.26775]. ### Limitations - **Prompt Length and Compute:** Each additional demonstration incurs a quasi-linear increase in input size and inference cost, with transformer attention scaling quadratically [2312.02520]. - **Modality Imbalance:** Long visual sequences can overwhelm shorter text outputs, requiring careful loss reweighting or auxiliary objectives [2312.02520]. - **Context Utilization:** Many current VLMs remain strongly text-driven even in multimodal contexts, failing to fully exploit visual cues from demonstrations [2510.24331]. - **Class/Task Generalization:** Current frameworks typically operate on single-class, single-task per query; true multi-task and multi-class ICL requires extension of output encoding or architectural modifications. - **Data and Sequence Limits:** Token limits of modern transformers (2048–8192 tokens) cap the number of demonstrable context pairs and restrict the complexity of few-shot adaptation [2403.13164]. ## 6. Directions for Enhancement and Future Research Several key avenues have emerged: - **Explicit Cross-Modal Fusion:** Training objectives and attention mechanisms that explicitly ground visual outputs in demonstration images, rather than relying solely on textual similarity, are central to further gains [2510.24331]. - **Dynamic Prompting and Curriculum Learning:** Any-shot, multi-turn instruction tuning with semantically coherent conversation structures improves both few-shot and zero-shot visual reasoning [2403.12736]. - **Hybrid Vector and Sequence Summarization:** Learnable in-context vectors (LIVE) that distill demonstration set knowledge into compact shift vectors reduce computation and maintain high performance, particularly in vision-language question answering [2406.13185]. - **Higher-Level Task Unification:** Ongoing work extends V-ICL paradigms to cross-task adaptation (e.g., from restoration to enhancement to segmentation) and to new modalities (e.g., 3D, audio, web), leveraging generalized token and transformer frameworks [2312.02520]. Overall, V-ICL defines a rapidly evolving research area at the intersection of vision, multimodal learning, and foundation model prompting, combining unified architectures, demonstration-aware adaptation, and principled retrieval/fusion to enable training-free and highly flexible visual reasoning across a spectrum of domains and task types. --- **References** - [2312.02520] Towards More Unified In-context Visual Understanding - [2304.04748] Exploring Effective Factors for Improving Visual In-Context Learning - [2312.06592] Flexible visual prompts for in-context learning in computer vision - [2603.26775] Learning to Select Visual In-Context Demonstrations - [2510.24331] What do vision-language models see in the context? Investigating multimodal in-context learning - [2403.13164] VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning - [2406.13185] LIVE: Learnable In-Context Vector for Visual Question Answering - [2403.12736] Towards Multimodal In-Context Learning for Vision & Language Models - [2504.18158] E-InMeMo: Enhanced Prompting for Visual In-Context Learning - [2605.31145] FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization Source: https://www.emergentmind.com/topics/visual-in-context-learning-v-icl