In-Context Visual Learning

• In-context visual learning is a paradigm where pretrained vision models rapidly adapt to new tasks using example-based prompts without changing model parameters.
• It employs diverse architectures such as grid-encoders, memory-augmented attention, and diffusion methods to support compositional reasoning and causal inference.
• Key components include prompt selection, fusion, and global ranking, which together boost performance in segmentation, detection, personalization, and other vision tasks.

In-context visual learning is a paradigm in which large vision or vision-LLMs rapidly adapt to new visual tasks by conditioning their predictions on a structured set of input–output examples ("prompts") without updating model parameters. This capability, analogous to in-context learning in LLMs, enables rapid task adaptation, compositional reasoning, causal inference, and even user-centric interaction in computer vision. This article surveys the principles, algorithms, and major research threads defining in-context visual learning as of early 2026, with emphasis on context selection, prompt design, compositionality, and the emergence of interactive and personalized visual ICL systems.

1. Foundations and Definitions

In-context visual learning (ICL) refers to the ability of large, pretrained vision or vision-LLMs to perform novel visual tasks by observing a small collection of example input–output pairs at inference time—without any parameter or weight updates (Zhang et al., 2023, Foster et al., 2023, Sheng et al., 2023, Gu et al., 2024, Seoh et al., 20 May 2025). Formally, given a support set $S = \{(x_i, y_i)\}_{i=1}^{k}$ and a query $x_q$, the model computes

$y_q = \Psi(S, x_q)$

where $\Psi$ is typically a frozen, pretrained backbone repurposed for the prompt-structured input modality. The prompt may consist of visual pairs (image–mask, image–box, image–edit, etc.), visual-textual pairs (image–caption), or their multimodal compositions. ICL in vision draws on advances in masked autoencoding, diffusion models, vision transformers, and large-scale visual-language pretraining.

Two critical properties distinguish in-context visual learning from traditional transfer or meta-learning:

• No weight updates at inference: All adaptation arises from conditioning on prompt examples rather than parameter optimization.
• Flexible task specification: Any visual mapping that can be formatted as (input, output) pairs is in principle eligible for ICL, supporting rapid adaptation to novel, user-defined objectives.

Deployments span semantic segmentation, detection, colorization, image translation, compositional VQA, personalized vision (user-specified new objects or relations), emotion understanding, and interactive editing (Sun et al., 2023, Jiang et al., 29 Sep 2025, Schmidt et al., 8 Apr 2026, Xiong et al., 17 Mar 2026, Nulli et al., 2024).

2. Algorithmic Frameworks and Model Architectures

A wide array of algorithmic skeletons for in-context visual learning have been developed, unified in their treatment of demonstration-based prompting but differing in backbone, tokenization, fusion method, and outputs.

2.1 Grid-Encoder Architectures

Image input–output pairs are arranged in a spatial grid or "canvas," often as a $2\times 2$ block where one query region is left blank. The model (e.g., MAE-VQGAN (Zhang et al., 2023, Sheng et al., 2023), Stable Diffusion (Oorloff et al., 13 Aug 2025)) inpaints the missing region based on the provided support. This grid-format is naturally extendable to multi-shot and is the basis for many state-of-the-art ICL pipelines (Sheng et al., 2023, Gu et al., 2024, Jiang et al., 29 Sep 2025).

2.2 Memory-Augmented Attention

For segmentation, video object segmentation models such as XMem treat the support set as a per-pixel memory of visual key/value pairs, allowing the query features to cross-attend over all support examples (Foster et al., 2023). This "global prompt memory" supports efficient multi-shot ICL and robust generalization to unseen classes.

2.3 Diffusion ICL

Diffusion backbones condition generative image outputs on prompt grids using modified attention schemes (e.g., self-attention cloning, cross-attention masking), enabling powerful analogy-based visual translation, editing, and personalization (Gu et al., 2024, Jiang et al., 29 Sep 2025). These approaches often require no fine-tuning and operate in a fully out-of-the-box (zero-shot ICL) regime.

2.4 Unified and Multimodal Transformers

Tokenizing images (via VQGAN) and text (via BPE) into a joint discrete vocabulary, decoder-only sparse transformers perform next-token prediction over interleaved visual/textual sequences, thus unifying image-to-image and image-to-text ICL in a single generative pipeline (Sheng et al., 2023).

2.5 Prompt Enhancement Modules

Learnable prompt perturbations, such as border-enriched additive cues, are trained (with the backbone frozen) to shift the prompt distribution and improve ICL robustness and accuracy for downstream tasks (Zhang et al., 2023, Zhang et al., 25 Apr 2025).

3. Prompt Selection, Fusion, and Global Ranking

The selection and combination of in-context examples are now recognized as dominant factors in ICL performance:

• Prompt Selection: Selecting prompts via pixel-level or semantic similarity (e.g., CLIP, DINO features) improves mIoU by up to +14 points over random, while learned or list-wise global ranking (Partial2Global, RH-Partial2Global) further narrows the gap to combinatorial optimum (Xu et al., 2024, Wu et al., 30 Sep 2025, Zhang et al., 2023, Sun et al., 2023). RL-based selection becomes strictly necessary for regression tasks with broad output distributions, actively enforcing diversity among prompt outputs (Lee et al., 24 Mar 2026).
• Prompt Fusion: Exhaustive fusion of multiple prompt arrangements (spatial layouts) or multi-branch, cross-attention-based architectures unlock complementary contextual cues, yielding new state of the art in challenging few-shot and open-domain benchmarks (Liao et al., 15 Jan 2026, Sun et al., 2023). Collaborative multi-group fusion (MULTI-VQGAN) outperforms both single-best and naive averaging approaches.
• Task- vs. Sample-level Search: For many standard benchmarks, a single prompt set identified at the task level suffices for near-optimal performance across all queries, substantially reducing deployment costs (Zhu et al., 15 Jan 2025).
• Counterfactual Retrieval: Actively selecting or composing counterfactual (attribute-intervened) demonstrations exposes decision boundaries and enables more causal, robust generalization for compositional and multimodal ICL (Xiong et al., 17 Mar 2026).

4. Applications: Segmentation, Recognition, Reasoning, and Personalization

In-context visual learning supports a spectrum of applications:

• Semantic and instance segmentation: Few-shot binary or multi-class segmentation via grid-formatted ICL achieves strong out-of-distribution generalization, even matching or surpassing meta-learning methods (Foster et al., 2023, Sun et al., 2023, Zhang et al., 25 Apr 2025, Zhang et al., 2023).
• Detection and keypoint localization: Prompting with bounding-box or heatmap outputs in the context enables competitive performance without direct task-specific fine-tuning (Oorloff et al., 13 Aug 2025).
• Colorization and edge detection: Unified ICL pipelines accommodate mask, color, and edge outputs by appropriate prompt construction and tokenization (Oorloff et al., 13 Aug 2025, Liao et al., 15 Jan 2026).
• Visual reasoning and compositionality: Assembling sequences of image–caption pairs into in-context prompts enhances compositional understanding, mitigates "bag-of-objects" bias, and supports chain-of-thought reasoning in VLMs (Nulli et al., 2024, Zhou et al., 2024, Sheng et al., 2023).
• Emotion recognition and visual sentiment: Retrieval of context-dependent, cluster-based label descriptions (e.g., EmoGist) sharply improves LVLM accuracy on nuanced, context-sensitive affect classification (Seoh et al., 20 May 2025).
• Personalized and interactive vision: Quad-grid ICL frameworks enable users to define new edit or transformation tasks via a single annotated example, which the model then generalizes to queries with user-specific transformations or explicit interaction cues (Schmidt et al., 8 Apr 2026, Jiang et al., 29 Sep 2025).

5. Evaluation Protocols, Benchmarks, and Quantitative Trends

Key benchmarks span Pascal-5$^i$ (segmentation), VOC 2012 (detection), Cityscapes (segmentation), ImageNet and ADE20K (colorization, super-resolution), CUB/Flowers (fine-grained classification), and multiple VQA and emotion understanding datasets (Foster et al., 2023, Sheng et al., 2023, Gu et al., 2024, Seoh et al., 20 May 2025, Xiong et al., 17 Mar 2026, Nulli et al., 2024).

Main performance metrics include:

Problem	Metric	Typical Gains from ICL-specific Methods
Segmentation	mIoU (↑)	+8–14 pt vs. random, +2–3 pt over SupPR
Detection	mIoU (↑)	+10–17 pt over non-enhanced prompts
Colorization	MSE, LPIPS (↓)	–0.05 MSE, –0.1 LPIPS (ensemble methods)
VQA	Acc, F1 (↑)	+6–7% over image-only or random retrieval
Compositional	Text Acc (↑)	+4–10% with compositional demonstration sets
Emotion	F1 (↑)	+8–13 pts using context-dependent label desc

ICL methods are generally task-agnostic and plug-and-play, admitting rapid extension to novel user-defined tasks and domains without re-training.

6. Limitations, Open Challenges, and Future Directions

Despite the versatility of in-context visual learning frameworks, current methods face several limitations:

• Data and class sensitivity: Learnable prompt perturbations are task- and class-specific; substantial in-domain data per class may be necessary to reach optimal performance (Zhang et al., 25 Apr 2025, Zhang et al., 2023).
• Resolution and detail: Some grid-based or VQGAN-tokenized outputs are limited to 256×256, constraining fine detail especially for super-resolution and precise localization (Schmidt et al., 8 Apr 2026).
• Single-pass/one-shot/linear context: Most ICL systems operate with a fixed set of context examples, lacking iterative or multi-round interaction/correction (Schmidt et al., 8 Apr 2026, Jiang et al., 29 Sep 2025).
• Prompt diversity and global ranking: Current selection/ranking is primarily similarity-based; more nuanced causality- or diversity-aware metrics, as well as holistic covering designs, are active research frontiers (Wu et al., 30 Sep 2025, Xiong et al., 17 Mar 2026).
• Cross-modal fusion: Unified multimodal ICL, especially for generative vision-LLMs, remains an ongoing challenge, with tokenization and block-sparse attention as one promising pathway (Sheng et al., 2023).
• Inference efficiency: While the cost of prompt search can be amortized at the task level, diffusion-based inference and multi-branch fusion incur significant runtime (Zhu et al., 15 Jan 2025, Liao et al., 15 Jan 2026, Oorloff et al., 13 Aug 2025).
• Causal and compositional understanding: Standard nearest-neighbor retrieval often selects correlated but non-causal exemplars; actively retrieving counterfactuals yields more robust generalization but requires attribute extraction and intervention in the pool (Xiong et al., 17 Mar 2026).

Open research areas include integrating multi-round human–AI interaction, efficient high-resolution generative backbones, globally optimal but scalable prompt selection, and more general causal, compositional, or multi-modal reasoning.

7. Broader Significance and Impact

In-context visual learning fundamentally changes the interface between users and visual models: new task definitions, personalized operations, and semantic reasoning are available "out of the box" through example-based prompting. As models and selection strategies improve, the paradigm is expected to further bridge semantic gaps, support personalized and user-driven visual workflows, and anchor computer vision models in genuine analogical, compositional, and causal learning regimes.

Key research lines demonstrate that careful retrieval, fusion, and prompt design—rather than parameter fine-tuning—are the main levers for rapid, robust, and flexible visual task adaptation in the current generation of vision and vision-language foundation models (Zhang et al., 2023, Foster et al., 2023, Sheng et al., 2023, Liao et al., 15 Jan 2026, Xu et al., 2024, Lee et al., 24 Mar 2026, Xiong et al., 17 Mar 2026, Jiang et al., 29 Sep 2025, Oorloff et al., 13 Aug 2025).