---
title: Multimodal In-Context Learning
url: https://www.emergentmind.com/topics/multimodal-in-context-learning
type: topic
---

# Multimodal In-Context Learning

Multimodal in-context learning (ICL) is the inference-time adaptation regime in which a pretrained large vision-language model conditions on a small set of image-text demonstrations, optionally together with an instruction, and produces an answer for a new query without any gradient update. In a common formulation, given demonstrations \(\{D_1,\dots,D_N\}\), an instruction \(Inst\), and a query \((\hat I,\hat Q)\), the model outputs \(\hat R\) as \(\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}\) [2505.17097]. Across recent work, the term covers both understanding and generation, including visual question answering, captioning, classification, reasoning, text-to-image generation, and unified image-text interaction. At the same time, the literature consistently characterizes the setting as fragile: performance is highly sensitive to retrieval, ordering, prompt construction, and internal attention dynamics, and the apparent ability to improve with more demonstrations does not necessarily imply effective multimodal integration [2505.17097].

## 1. Formal definition and scope

A generic multimodal ICL episode consists of a support set \(S=\{(x_i,y_i)\}_{i=1}^k\), an optional instruction \(I\), and a query \(x^*\), with the model estimating \(p_\theta(y^*\mid x^*,I,S)\) in a single feed-forward or autoregressive pass [2403.13164]. In vision-language settings, \(x_i\) may be an image, text, or an interleaved image-text input, while \(y_i\) may be text or, in unified models, an image [2403.13164]. Another common formalization writes the context as \(C=((I_i,T_i,R_i))_{i\in D_C}\) and the query as \(Q=(I_Q,T_Q)\), with output \(O=\mathrm{LMM}(C,(I_Q,T_Q))\) [2404.15736]. These formulations agree on the essential property: adaptation is induced by the prompt alone.

The scope of the field has expanded from conventional few-shot VQA and captioning to broader “vision-language ICL” settings that test perception, fast concept binding, rule induction, interleaving, long-context behavior, and text-to-image generation [2403.13164]. Unified formulations further emphasize that images and text share the same autoregressive “canvas,” which allows a single model to accept arbitrary mixes of modalities within a common interface \(y^* \sim p_\theta(y\mid D,x^*,q^*)\) [2603.24690]. This broadened scope matters because early evaluations based primarily on VQA and captioning were later argued to under-test the distinctive strengths and limitations of in-context learning in multimodal systems [2403.13164].

The distinction from text-only ICL is not merely the presence of visual tokens. Multimodal ICL requires the model to construct and transfer a task mapping under cross-modal grounding constraints. Several papers therefore frame the problem not simply as “few-shot prompting with images,” but as the joint inference of task structure, relevant evidence, and cross-example regularities from interleaved visual and textual context [2505.17097].

## 2. Architectural substrates and the emergence of multimodal ICL

Most modern systems inherit a common architectural pattern: a visual encoder produces image features, a projector or adapter maps those features into the language-model space, and an autoregressive decoder processes the combined sequence. Decoder-based vision-language models commonly insert projected image embeddings where special image tags appear in the prompt, but standard alignment pipelines historically emphasized single-image, instruction-following data rather than explicit multi-shot, multi-image episodes [2403.12736]. This training mismatch became a central explanation for weak few-shot behavior.

Generative interleaved models provided an early indication that scaling and unified autoregressive training can induce stronger multimodal ICL. Emu2 connects a pretrained EVA-02-CLIP-E-plus visual encoder, a LLaMA-33B-initialized decoder-only transformer, and an SDXL-base visual decoder in a single autoregressive loop, treating text tokens and continuous visual embeddings as elements of one sequence [2312.13286]. Emu2 is trained with a unified objective combining text cross-entropy and image embedding regression, and at 37B parameters it reports 67.8% on VQAv2 in the 8-shot setting, exceeding Flamingo 80B at 65.6% and IDEFICS 80B at 64.8% [2312.13286]. The same work reports emergent “visual prompting” and “object-grounded generation,” indicating that in-context adaptation can extend beyond standard recognition-style tasks [2312.13286].

However, later studies complicate the interpretation of such gains. A systematic analysis of seven VLM checkpoints across four architectural families found that training on image-text interleaved data improves shot scaling in captioning, but “does not imply effective integration of visual and textual information from demonstration examples” [2510.24331]. The same study reports that instruction tuning improves instruction-following and zero-shot performance while sometimes reducing reliance on demonstrations; for Idefics2, the instruction-tuned variant underperforms the base variant once demonstrations are introduced, losing up to \(\sim 20\) CIDEr-D points at higher \(k\) [2510.24331]. This suggests that architectural support for interleaving and stronger instruction alignment are neither equivalent to, nor sufficient for, robust multimodal ICL.

A complementary line of work argues that explicit ICL exposure during training is itself a missing ingredient. A multi-turn curriculum that converts single-turn visual conversations into any-shot episodes reports a significant 21.03% gain on one held-out ICL task and approximately 11.3% average improvement over strong VLM baselines [2403.12736]. The result strengthens the view that multimodal ICL is partly a learned interface behavior rather than a guaranteed byproduct of general multimodal pretraining.

## 3. Empirical regularities and recurrent bottlenecks

A dominant empirical theme is modality asymmetry. A broad study of open-source multimodal ICL found that, whenever text is present, images often play a minor role: in VQA settings, removing question text costs 3.5 percentage points on average and randomizing it costs 9.5 points, whereas removing or shuffling images loses only approximately \(1\)–\(1.5\) points [2404.15736]. The same work reports that in classification, text alone matches zero-shot behavior at approximately \(+0.5\) points, while captioning benefits substantially more from richer textual context, with images contributing an additional \(+20\) CIDEr after caption text already provides \(+31\) CIDEr over zero-shot [2404.15736]. Another attention-centered study reaches a similar conclusion: even when models show positive shot scaling, black-image and no-image ablations often have minor effects, and attention remains concentrated on textual prefixes and recent words rather than demonstration images [2510.24331].

A second regularity is instability with more context. Several works report that additional demonstrations do not guarantee improvement and can even reduce performance. In controlled evaluation, some models plateau or collapse as shot count grows; LLaVA v1.5 is reported to fall to near-zero captioning performance by \(k=8\) in one benchmark suite [2510.24331]. On broader multimodal tasks, performance often saturates or degrades once total token count approaches the pretrained context length, and the per-shot gain \(\Delta_k=\mathrm{Acc}(k)-\mathrm{Acc}(k-1)\) frequently becomes small or negative beyond a small number of shots [2403.13164]. This non-monotonicity is one reason recent work treats “more examples” as an unreliable design principle in multimodal settings.

A third bottleneck is bias induced by sequence structure. Retrieval-based strategies can improve raw performance, but analyses indicate that part of the benefit comes from retrieving examples whose outputs are already close to the target, rather than from deeper multimodal reasoning [2404.15736]. Recency effects are particularly pronounced: generalized linear modeling shows stronger similarity-performance coefficients for later demonstrations, and exact-match analysis finds that the last demonstration’s response is reproduced approximately 12% of the time in VQA and up to 24% of the time in 4-shot captioning [2404.15736]. These results align with later methods that explicitly counter positional concentration or reorder demonstrations.

A fourth bottleneck concerns internal transfer rather than mere perception. A mechanistic study comparing text-only and multimodal ICL under identical task formulations reports that multimodal models perform comparably to text-only ICL in zero-shot settings but degrade significantly under few-shot demonstrations [2604.13403]. For Qwen2.5-VL-7B, the paper gives 38.8% for text-only versus 45.2% for multimodal in zero-shot, but 88.4% versus 63.6% in 4-shot; for Gemma-3-12B, the 4-shot figures are 91.4% versus 72.9% [2604.13403]. The analysis decomposes multimodal ICL into task mapping construction and task mapping transfer, and argues that current models often ground demonstrations in mid-layers but fail to transfer the learned mapping to the query at the appropriate stage [2604.13403]. A plausible implication is that multimodal ICL fails less because models cannot “see” and more because they cannot preserve and reuse cross-modal task structure across layers.

## 4. Context construction, retrieval, ordering, and task mapping

A large body of work treats context construction as the principal control surface for multimodal ICL. A comprehensive study of six VLLMs and twenty strategies across retrieval, ordering, and prompt construction reports three robust findings: a true multimodal retriever is necessary, intra-demonstration ordering matters more than inter-demonstration ordering, and introductory instructions improve task comprehension [2410.20482]. In that study, a multimodal retriever based on BridgeTower with cosine similarity improves average score by approximately 3.8% relative to zero-shot and single-modality retrieval; placing vision first within each demonstration (\(IOP^{vtt}\)) yields approximately \(2.0\)–\(3.0\) point gains over text-first variants; and a brief introductory instruction \(\mathcal{I}_{intro}\) adds a stable \(+1.2\%\) on average [2410.20482]. By contrast, switching between random, similar-first, and similar-last inter-demonstration orderings changes performance by less than \(0.5\) point on average [2410.20482].

Other work elevates “task mapping” from an interpretive metaphor to an explicit design principle. In this view, each demonstration \((I_i,Q_i,R_i)\) defines a local mapping \(f_i:(I_i,Q_i)\rightarrow R_i\), while the full context induces a global mapping \(\hat f:(\hat I,\hat Q)\rightarrow \hat R\) [2505.17098]. Sequence quality can then be assessed by how well these local mappings align with one another and with the query. TACO operationalizes this idea with a lightweight transformer decoder and task-aware attention, and introduces two sequence-level metrics: the Disruption Gap \(\Delta\), measuring sensitivity to replacing one demonstration with a nearest neighbor, and Order Sensitivity \(\sigma\), the standard deviation of accuracy over permutations of the same demonstration set [2505.17098]. On nine datasets and five LVLMs, TACO raises 4-shot VQAv2 average accuracy from 64.13% to 66.75%, OK-VQA from 58.33% to 61.54%, and MSCOCO captioning CIDEr from 118.27 to 119.47 [2505.17098].

SabER develops a closely related perspective with a decoder-only transformer that autoregressively selects and arranges in-context demonstrations from a library, using task-aware attention and a task guider embedding to refine task mapping hierarchically [2503.04839]. Across five LVLMs and nine benchmarks, its reported improvements include VQAv2 from 57.86 to 64.74, VizWiz from 41.94 to 50.77, and OK-VQA from 49.89 to 57.77 in the 4-shot setting [2503.04839]. These models differ in detail, but both treat demonstration configuration as a learned sequence-construction problem rather than a static similarity search.

Reinforcement learning has also been applied to the same problem. EE-ICL formulates demonstration selection as a sequential decision process over multimodal candidates and learns a policy by combining stochastic beam search with policy-gradient updates [2506.09473]. On four VQA datasets with \(m=2\) demonstrations, it reports VQAScore improvements over similarity-based retrieval, including \(+4.1\) on OKVQA and \(+4.9\) on VizWiz, with an overall average gain of \(+1.6\) [2506.09473]. These results support a broader conclusion already implicit in task-mapping work: individual demonstrations cannot be evaluated independently, because their joint utility depends on redundancy, complementarity, and compatibility with the query.

## 5. Internal interventions, compression, and virtual-context methods

Because prompt engineering alone leaves the model’s internal mechanism untouched, a later wave of work intervenes directly in attention, residual streams, or virtual context representations. The resulting methods differ in whether they are training-free, parameter-efficient, or full learned modules, but they share a common objective: to preserve or approximate the effect of demonstrations without relying exclusively on raw concatenation.

| Method | Core mechanism | Reported effect |
|---|---|---|
| CAMA [2505.17097] | Training-free modulation of attention logits with a query-ICD joint affinity score \(J_j\) and positional context factor \(P_j\) | Average gain of \(+2.53\%\) over standard attention across four LVLMs and six benchmarks |
| M2IV [2504.04633] | Layer-wise learnable in-context vectors injected into MHA and MLP branches, stored in VLibrary | Average accuracy gain of \(3.74\%\) over ICL with the same shot count |
| HiFICL [2603.12760] | Virtual key-value pairs with low-rank factorization, trained end-to-end as context-aware PEFT | On Idefics2-8B-base: 72.08% VQAv2, 59.56% OK-VQA, 1.2951 COCO CIDEr |
| AIM [2406.07588] | Aggregates image information into text-side fused virtual tokens for frozen MLLMs | Drops visual-token burden to a token-ratio of approximately 8% for QWen-VL and 4% for LLaVA-Next |
| MTV [2406.15334] | Compresses many-shot demonstrations into attention-head activations and head locations | Encodes 400 examples into \(\sim 100\) attention heads; Qwen-VL reaches 45.6% on VizWiz versus 44.3% for 8-shot ICL |

CAMA is representative of inference-stage attention recalibration. It starts from a theoretical analysis of multimodal attentional dynamics and identifies three limitations of standard attention: inadequate intra-ICD cross-modal alignment, imprecise query-sample-guided attention, and neglected inter-ICD influences [2505.17097]. Its intervention adds two log-bias terms to the raw attention logits,
\[
L_{CAMA_j}^{(t)}=L_{raw_j}^{(t)}+\lambda_J\log(J'_j+\varepsilon)+\lambda_P\log(P_j+\varepsilon),
\]
where \(J'_j\) is a normalized query-ICD affinity derived from CLIP-based fused embeddings and \(P_j\) downweights positional bias via \(\bigl(\tfrac{N-pos_j+1}{N}\bigr)^k\) [2505.17097]. The method is explicitly training-free and plug-and-play.

A second family replaces explicit demonstrations with compact learned surrogates. M2IV learns layer-wise vectors \(\{(\alpha_l^a,v_l^a,\alpha_l^m,v_l^m)\}\) that emulate the residual contribution of demonstrations at the MHA and MLP branches, using a self-distillation objective with mimicry, synergy, and supervised losses [2504.04633]. AIM instead reads each image-text demonstration with a frozen MLLM, extracts hidden states on the text positions, projects them into fused virtual tokens, and reuses those tokens as text-like substitutes for the original multimodal demonstrations [2406.07588]. HiFICL makes a stronger theoretical claim: the influence of demonstrations in self-attention can be decomposed exactly into a query-dependent mixture of ordinary self-attention and context values, motivating virtual low-rank key-value pairs as a high-fidelity approximation rather than a heuristic shift vector [2603.12760].

A third family targets many-shot scaling and unified multimodal regimes. MTV extracts Multimodal Task Vectors by averaging hidden activations over many batches and identifying attention-head locations where these activations should be injected during inference, thereby encoding many examples into a fixed set of head activations with no extra context length at test time [2406.15334]. UniICL addresses a related problem at the level of a unified model spanning both understanding and generation: its Context-Adaptive Prototype Modulator decouples demonstration encoding, builds an inter-demo prototype bank, and injects adaptive gates into backbone layers, yielding understanding peak 80.9 versus BAGEL’s 72.7 and ICL efficiency \(+9.7\) on UniICL-Bench [2603.24690].

Analysis-driven methods also attempt to repair specific internal failures exposed by interpretability work. Mapping-Guided Inference estimates a peak grounding layer \(\ell^*\) by minimizing label-to-image attention entropy across demonstrations, treats the resulting attention pattern as an estimated task mapping \(\hat M\), and then amplifies the query’s attention to salient demonstration image tokens in later layers [2604.13403]. The gains are deliberately modest—such as 69.09% to 70.17% on Outlier Detection and 48.13% to 48.17% on OK-VQA—but consistent across settings [2604.13403]. This modesty is itself informative: it suggests that inference-time steering can help, yet deeper architectural or training changes may be required to close the multimodal ICL gap.

## 6. Evaluation regimes, misconceptions, and open directions

A recurring criticism of early evaluation is that conventional VQA and captioning benchmarks do not isolate the capacities that make in-context learning distinctive. VL-ICL Bench was introduced precisely to expand coverage, with eight tasks spanning fast concept binding, fine-grained perception, rule induction, interleaving, long-context learning, and latent-variable induction in generation [2403.13164]. Its results show that even strong models struggle on perception-heavy or context-heavy tasks: in the 2-shot case, GPT-4V reaches 48% on Fast Open MiniImageNet, 30% on CLEVR Count Induction, 84% on Operator Induction, and 50% on TextOCR, while interleaved or long-context tasks remain harder [2403.13164]. This broadening of evaluation reframes multimodal ICL as more than few-shot VQA.

Unified evaluation pushes the taxonomy further. UniICL organizes demonstration roles into six levels—Perception, Imitation, Conception, Deduction, Analogy, and Discernment—and builds UniICL-760K with 766,868 eight-shot episodes across 15 subtasks, plus a 1,250-episode UniICL-Bench covering \(k\in\{0,1,2,4,8\}\) [2603.24690]. This taxonomy is consequential because it makes explicit that demonstrations do qualitatively different kinds of work across tasks. A plausible implication is that “multimodal ICL” is not a single capability but a family of adaptation behaviors whose failure modes differ by cognitive demand.

One common misconception is that positive shot scaling proves genuine multimodal learning from demonstrations. Multiple studies dispute that inference. Interleaved-data training improves shot scaling but often leaves attention predominantly text-centric [2510.24331], while broader analyses argue that multimodal ICL frequently reduces to “text-only ICL + superficial image token” whenever text is available [2404.15736]. Another misconception is that better retrieval alone solves the problem. ContextNav, an agentic framework for retrieval, denoising, structural alignment, and graph-based workflow orchestration, reports an average ICL gain of 16.8% across six MLLMs versus 7.6% for the best prior baseline, but its own design centers on noise-robust contextualization rather than simple nearest-neighbor search [2510.04560]. The result indicates that scalable retrieval must be coupled with curation and feedback.

Recent work increasingly frames the next stage as a reasoning problem. MMInduction defines an “inductive gap” \(G=A-I\) between answer accuracy and induction accuracy, arguing that models often produce correct answers from flawed reasoning while failing to extract the underlying rule from demonstrations [2605.02378]. Its response combines similarity-based visual token compression, dynamic attention rebalancing, and an explicit inductive-deductive chain-of-thought, followed by supervised fine-tuning and reinforcement learning with verifiable rewards [2605.02378]. In a different vein, synthetic-mechanistic analysis shows that multimodal ICL can emerge from induction-style circuits analogous to those studied in text-only transformers, with “previous-token” and “induction” heads remaining causally important across modalities; the same study reports that Rotary Position Embeddings increase the data complexity threshold for ICL [2601.20796]. Taken together, these lines of work suggest that future progress will likely depend on jointly addressing representation alignment, context management, and circuit-level transfer.

The field therefore stands at a transition point. Early progress established that large multimodal models can, under some conditions, adapt from examples in context. Subsequent work clarified that the behavior is often brittle, text-dominated, and highly contingent on sequence design. Current research is moving toward a more mechanistic and systems-level conception in which multimodal ICL is shaped by task mapping, cross-modal alignment, internal attention routing, virtualized context representations, and increasingly diagnostic evaluation suites. Whether future models will turn these ingredients into robust cross-modal induction remains an open question, but the literature already converges on one negative conclusion: raw interleaving of images and text is not, by itself, sufficient.

Source: https://www.emergentmind.com/topics/multimodal-in-context-learning