Multimodal In-Context Learning (MICL)

• MICL is a technique that enables models to rapidly adapt to new multimodal tasks by conditioning on interleaved image, text, and other modality examples without explicit parameter updates.
• Recent advances highlight specialized architectures that fuse image and text tokens, supporting both fine-grained perception and high-level reasoning for tasks like VQA and document recognition.
• Empirical studies stress that effective demonstration selection and context formatting can boost few-shot performance by up to 18%, while mitigating biases such as recency effects.

Multimodal In-Context Learning (MICL) refers to the ability of vision-language or multimodal models—typically built atop LLM backbones—to rapidly adapt to new multimodal tasks at inference by conditioning on a handful of interleaved image, text, and potentially other modality examples, without explicit parameter updates or gradient steps. Unlike conventional supervised learning, MICL enables models to generalize across diverse and previously unseen multimodal prompt structures, provided task information and demonstration exemplars as part of the input. Recent research demonstrates that while the promise of MICL is significant—especially for general-purpose visual reasoning, open-vocabulary recognition, and rapid task adaptation—effective and truly multimodal in-context learning requires advances in model architectures, context schemes, demonstration selection strategies, and evaluation methodology.

1. Model Architectures and Processing Paradigms

Recent advances in MICL depart from simple concatenation of image and text features, instituting specialized architectures to fuse and interleave modalities. MMICL (Zhao et al., 2023) exemplifies this with its balanced design: image features are extracted via a vision encoder (e.g., ViT), projected into the language embedding space, and inserted as unique “image declaration” tokens directly into the text stream. These image tokens are interleaved with associated textual prompts before passing into the LLM backbone. The image–text interleaving pattern is formalized as

$$\mathbf{I}_i = \left( \{\mathbf{x}_1, \ldots, \mathbf{x}_k\},\, \mathbf{q}_i,\, \mathbf{a}_i \right)$$

where $\{\mathbf{x}_1, \ldots, \mathbf{x}_k\}$ are image declarations, $\mathbf{q}_i$ is the question/instruction, and $\mathbf{a}_i$ is the answer. The unified multimodal representation supports both perception (fine-grained object and scene parsing) and higher-level reasoning.

Generative models like Emu2 (Sun et al., 2023) and decoder-only designs (e.g., SabER (Li, 5 Mar 2025)) further generalize MICL by treating entire multimodal sequences (text, image tokens, region or coordinate tokens) autoregressively, permitting both input and output to span mixed modalities.

Approaches such as MMICT (Chen et al., 2023) and M²IXT (Chen et al., 2023) introduce additional learnable adaptation modules (e.g., Multi-Modal Hub, lightweight context-tuning modules) that selectively align and fuse contextual features from demonstrations, enabling parameter-efficient in-context adaptation across diverse vision-language backbones.

2. Context Scheme and Demonstration Formatting

A core challenge in MICL is the precise formatting and alignment of multimodal demonstrations within the input sequence. MMICL (Zhao et al., 2023)’s context scheme explicitly declares each image in the input via dedicated tokens (e.g., “[IMG₁]”), which are referred to in the text and co-embedded with image features. Interconnected image groupings (cropped objects, video frames) are encoded to support relational reasoning.

Unified in-context learning formats organize each episode as $N$ few-shot exemplars prepended to the query, where each demonstration is an image–text–answer tuple. All downstream tasks—including captioning, VQA, and reasoning—are recast into this uniform in-context “teach-by-example” style.

New benchmarks and curricula (e.g., the multi-turn conversation approach (Doveh et al., 19 Mar 2024)) highlight the empirical value of semantically coherent in-context sequences, where all demonstrations in a batch share task type or attribute (e.g., all about color, counting, or object location). Such alignment accelerates the emergence and evaluation of in-context capabilities.

3. Demonstration Selection and Retrieval Strategies

Retrieval and configuration of demonstrations critically influence MICL performance. Empirical studies (Chen et al., 2023, Luo et al., 19 Apr 2024, Baldassini et al., 24 Apr 2024, Wu et al., 22 May 2025) show that:

• Textual content of the demonstrations is the dominant contributor to successful adaptation for most tasks.
• Visual tokens in demonstrations (e.g., image references) have minor direct effect in prevailing architectures, due to masked cross-attention mechanisms that restrict visual flow.
• Demo selection stratagems that blend both visual and textual similarities (MMICES) (Chen et al., 2023), or leverage supervised retrievers trained to maximize output likelihood (Luo et al., 19 Apr 2024), yield superior results compared to random or visual-only methods.
• Demonstration order exerts a recency bias (Rieff et al., 26 Jun 2025), where models copy the answer or style of the last (most recent) demonstration, implicating prompt engineering and ordering in robust performance.
• Balancing or intentionally controlling label (output) distributions within demonstrations mitigates predictive bias, as demonstrated in sentiment analysis (Wu et al., 22 May 2025).

For tasks with strong visual groundings (e.g., key information extraction from documents, novel concept learning), vision-driven or dual-modality retrieval is vital; for most standard VQA and classification, text-prioritized selection suffices (Xu et al., 1 Jul 2024).

4. Performance Findings, Limitations, and Key Advances

MMICL sets state-of-the-art zero-shot and few-shot performance on complex multi-modal benchmarks (MME, MMBench) (Zhao et al., 2023). Emu2 achieves leading results on VQAv2, OKVQA, VizWiz, TextVQA, both in understanding and generative settings (Sun et al., 2023). Lightweight modules (e.g., M²IXT (Chen et al., 2023)) demonstrate that parameter-efficient context integration can produce $18\%$ or higher relative few-shot improvement over strong baselines, rivaling much larger models.

However, studies across multiple benchmarks reveal structural limitations (Chen et al., 2023, Baldassini et al., 24 Apr 2024, Rieff et al., 26 Jun 2025, Chen et al., 21 Jul 2025):

• Most MLLMs remain essentially text-driven; multishot demonstration images in the prompt are routinely ignored for VQA tasks.
• Performance boosts from “advanced” retrieval (RICES, MMICES) can be matched by simple majority voting or copy-based mechanisms; the shot value often comes from “copying” answers, not true task learning (Baldassini et al., 24 Apr 2024).
• Recency bias and sensitivity to irrelevant (“noisy”) in-context examples are prevalent, especially in safety-critical domains (medical imaging (Rieff et al., 26 Jun 2025)), where a misplaced example degrades accuracy up to $9.5\%$.
• MICL performance seldom generalizes to tasks where true image-to-text integration is required. Dedicated datasets such as TrueMICL (Chen et al., 21 Jul 2025) show that unless specifically fine-tuned (e.g., DARA), models fail to perform genuine visual adaptation.

5. Mitigating Bias and Enhancing True Multimodal Learning

Recent innovations explicitly address the visual neglect and shortcut learning biases:

• Attention reallocation strategies (e.g., DARA (Chen et al., 21 Jul 2025)) rebalance the effect of visual and textual tokens in decoder attention, encouraging the model to process images in demonstrations.
• Fine-grained, task-aware attention and demonstration selection (as in SabER (Li, 5 Mar 2025)) ensure proper task mapping and context-dependency, avoiding modality or response shortcuts.
• Adaptive context mechanisms (feedback integration, category guidelines) (Liu et al., 21 Jun 2025) further support domain-specific adaptation, as in histopathology report generation.

Despite these advances, failure to process novel input-label mappings and inadequacies in learning from unseen multimodal prompts remain open challenges. Empirical studies consistently underline the value of principled demonstration configuration (semantics, order, content balance) in maximizing MICL potential.

6. Specialized and Domain Applications

MICL powers several application domains:

• Real-world dialogue and retrieval: Handling multistep spatial/temporal reasoning over interleaved image–text threads (Zhao et al., 2023).
• Document and script recognition: Open-vocabulary symbol classification in unseen scripts and alphabets (Simon et al., 9 Apr 2025).
• Biomedical and clinical reporting: Automated histopathology and visual reasoning with peer-contextual adaptation (Liu et al., 21 Jun 2025), SMMILE for medical benchmarking (Rieff et al., 26 Jun 2025).
• Sentiment analysis: Replacing supervised fine-tuning through task-optimized demonstration sets, achieving $15.9\%$ absolute accuracy gains over zero-shot pipelines (Wu et al., 22 May 2025).
• Continual multimodal learning and lifelong adaptation: Domain- and task-type-shifting with minimized forgetting via specialized modules (e.g., PTGM, IKD; (Pian et al., 17 Dec 2024)).

7. Open Research Questions and Future Directions

Several frontiers in MICL remain open:

• Architecture design. Can improved cross-modal integration and hierarchical attention amplify true visual context utilization?
• Benchmarking. How to build evaluation datasets (like TrueMICL) that distinguish between text imitation and genuine multimodal adaptation, crucial for real-world deployment?
• Modality extension. Inclusion of audio, video, and structured data in context (beyond image and text) is only beginning to be addressed (Pian et al., 17 Dec 2024).
• Adaptive and robust retrieval. Enhanced, possibly task-adaptive, supervised retrieval mechanisms are needed to mitigate bias and maximize the task relevance of demonstrations.
• Interpretability and bias analysis. Systematic paper of shortcut learning, recency effects, and distributional biases to inform better in-context representation and inference strategies.

Multimodal in-context learning is maturing towards systems that can generalize and adapt akin to human learning-by-example. Realizing the full potential of MICL demands advances at every stage—from architecture and data to evaluation—and mandates careful attention to the nuances and pitfalls of multimodal integration and prompt configuration.