Multimodal In-Context Learning

• Multimodal In-Context Learning is a framework where vision-language models adapt from a few image-text demonstrations at inference without gradient updates.
• It supports diverse tasks such as visual question answering, captioning, and rule induction, while its performance is highly sensitive to prompt design and demonstration ordering.
• Innovations like virtual-context methods and task mapping strategies aim to overcome bottlenecks in cross-modal integration and stability, improving few-shot performance.

Multimodal in-context learning (ICL) is the inference-time adaptation regime in which a pretrained large vision-LLM conditions on a small set of image-text demonstrations, optionally together with an instruction, and produces an answer for a new query without any gradient update. In a common formulation, given demonstrations $\{D_1,\dots,D_N\}$, an instruction $Inst$, and a query $(\hat I,\hat Q)$, the model outputs $\hat R$ as $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$ (Li et al., 21 May 2025). Across recent work, the term covers both understanding and generation, including visual question answering, captioning, classification, reasoning, text-to-image generation, and unified image-text interaction. At the same time, the literature consistently characterizes the setting as fragile: performance is highly sensitive to retrieval, ordering, prompt construction, and internal attention dynamics, and the apparent ability to improve with more demonstrations does not necessarily imply effective multimodal integration (Li et al., 21 May 2025).

1. Formal definition and scope

A generic multimodal ICL episode consists of a support set $S=\{(x_i,y_i)\}_{i=1}^k$, an optional instruction $I$, and a query $x^*$, with the model estimating $p_\theta(y^*\mid x^*,I,S)$ in a single feed-forward or autoregressive pass (Zong et al., 2024). In vision-language settings, $x_i$ may be an image, text, or an interleaved image-text input, while $Inst$0 may be text or, in unified models, an image (Zong et al., 2024). Another common formalization writes the context as $Inst$1 and the query as $Inst$2, with output $Inst$3 (Baldassini et al., 2024). These formulations agree on the essential property: adaptation is induced by the prompt alone.

The scope of the field has expanded from conventional few-shot VQA and captioning to broader “vision-language ICL” settings that test perception, fast concept binding, rule induction, interleaving, long-context behavior, and text-to-image generation (Zong et al., 2024). Unified formulations further emphasize that images and text share the same autoregressive “canvas,” which allows a single model to accept arbitrary mixes of modalities within a common interface $Inst$4 (Xu et al., 25 Mar 2026). This broadened scope matters because early evaluations based primarily on VQA and captioning were later argued to under-test the distinctive strengths and limitations of in-context learning in multimodal systems (Zong et al., 2024).

The distinction from text-only ICL is not merely the presence of visual tokens. Multimodal ICL requires the model to construct and transfer a task mapping under cross-modal grounding constraints. Several papers therefore frame the problem not simply as “few-shot prompting with images,” but as the joint inference of task structure, relevant evidence, and cross-example regularities from interleaved visual and textual context (Li et al., 21 May 2025).

2. Architectural substrates and the emergence of multimodal ICL

Most modern systems inherit a common architectural pattern: a visual encoder produces image features, a projector or adapter maps those features into the language-model space, and an autoregressive decoder processes the combined sequence. Decoder-based vision-LLMs commonly insert projected image embeddings where special image tags appear in the prompt, but standard alignment pipelines historically emphasized single-image, instruction-following data rather than explicit multi-shot, multi-image episodes (Doveh et al., 2024). This training mismatch became a central explanation for weak few-shot behavior.

Generative interleaved models provided an early indication that scaling and unified autoregressive training can induce stronger multimodal ICL. Emu2 connects a pretrained EVA-02-CLIP-E-plus visual encoder, a LLaMA-33B-initialized decoder-only transformer, and an SDXL-base visual decoder in a single autoregressive loop, treating text tokens and continuous visual embeddings as elements of one sequence (Sun et al., 2023). Emu2 is trained with a unified objective combining text cross-entropy and image embedding regression, and at 37B parameters it reports 67.8% on VQAv2 in the 8-shot setting, exceeding Flamingo 80B at 65.6% and IDEFICS 80B at 64.8% (Sun et al., 2023). The same work reports emergent “visual prompting” and “object-grounded generation,” indicating that in-context adaptation can extend beyond standard recognition-style tasks (Sun et al., 2023).

However, later studies complicate the interpretation of such gains. A systematic analysis of seven VLM checkpoints across four architectural families found that training on image-text interleaved data improves shot scaling in captioning, but “does not imply effective integration of visual and textual information from demonstration examples” (Santos et al., 28 Oct 2025). The same study reports that instruction tuning improves instruction-following and zero-shot performance while sometimes reducing reliance on demonstrations; for Idefics2, the instruction-tuned variant underperforms the base variant once demonstrations are introduced, losing up to $Inst$5 CIDEr-D points at higher $Inst$6 (Santos et al., 28 Oct 2025). This suggests that architectural support for interleaving and stronger instruction alignment are neither equivalent to, nor sufficient for, robust multimodal ICL.

A complementary line of work argues that explicit ICL exposure during training is itself a missing ingredient. A multi-turn curriculum that converts single-turn visual conversations into any-shot episodes reports a significant 21.03% gain on one held-out ICL task and approximately 11.3% average improvement over strong VLM baselines (Doveh et al., 2024). The result strengthens the view that multimodal ICL is partly a learned interface behavior rather than a guaranteed byproduct of general multimodal pretraining.

3. Empirical regularities and recurrent bottlenecks

A dominant empirical theme is modality asymmetry. A broad study of open-source multimodal ICL found that, whenever text is present, images often play a minor role: in VQA settings, removing question text costs 3.5 percentage points on average and randomizing it costs 9.5 points, whereas removing or shuffling images loses only approximately $Inst$7–$Inst$8 points (Baldassini et al., 2024). The same work reports that in classification, text alone matches zero-shot behavior at approximately $Inst$9 points, while captioning benefits substantially more from richer textual context, with images contributing an additional $(\hat I,\hat Q)$0 CIDEr after caption text already provides $(\hat I,\hat Q)$1 CIDEr over zero-shot (Baldassini et al., 2024). Another attention-centered study reaches a similar conclusion: even when models show positive shot scaling, black-image and no-image ablations often have minor effects, and attention remains concentrated on textual prefixes and recent words rather than demonstration images (Santos et al., 28 Oct 2025).

A second regularity is instability with more context. Several works report that additional demonstrations do not guarantee improvement and can even reduce performance. In controlled evaluation, some models plateau or collapse as shot count grows; LLaVA v1.5 is reported to fall to near-zero captioning performance by $(\hat I,\hat Q)$2 in one benchmark suite (Santos et al., 28 Oct 2025). On broader multimodal tasks, performance often saturates or degrades once total token count approaches the pretrained context length, and the per-shot gain $(\hat I,\hat Q)$3 frequently becomes small or negative beyond a small number of shots (Zong et al., 2024). This non-monotonicity is one reason recent work treats “more examples” as an unreliable design principle in multimodal settings.

A third bottleneck is bias induced by sequence structure. Retrieval-based strategies can improve raw performance, but analyses indicate that part of the benefit comes from retrieving examples whose outputs are already close to the target, rather than from deeper multimodal reasoning (Baldassini et al., 2024). Recency effects are particularly pronounced: generalized linear modeling shows stronger similarity-performance coefficients for later demonstrations, and exact-match analysis finds that the last demonstration’s response is reproduced approximately 12% of the time in VQA and up to 24% of the time in 4-shot captioning (Baldassini et al., 2024). These results align with later methods that explicitly counter positional concentration or reorder demonstrations.

A fourth bottleneck concerns internal transfer rather than mere perception. A mechanistic study comparing text-only and multimodal ICL under identical task formulations reports that multimodal models perform comparably to text-only ICL in zero-shot settings but degrade significantly under few-shot demonstrations (Wang et al., 15 Apr 2026). For Qwen2.5-VL-7B, the paper gives 38.8% for text-only versus 45.2% for multimodal in zero-shot, but 88.4% versus 63.6% in 4-shot; for Gemma-3-12B, the 4-shot figures are 91.4% versus 72.9% (Wang et al., 15 Apr 2026). The analysis decomposes multimodal ICL into task mapping construction and task mapping transfer, and argues that current models often ground demonstrations in mid-layers but fail to transfer the learned mapping to the query at the appropriate stage (Wang et al., 15 Apr 2026). A plausible implication is that multimodal ICL fails less because models cannot “see” and more because they cannot preserve and reuse cross-modal task structure across layers.

4. Context construction, retrieval, ordering, and task mapping

A large body of work treats context construction as the principal control surface for multimodal ICL. A comprehensive study of six VLLMs and twenty strategies across retrieval, ordering, and prompt construction reports three robust findings: a true multimodal retriever is necessary, intra-demonstration ordering matters more than inter-demonstration ordering, and introductory instructions improve task comprehension (Qin et al., 2024). In that study, a multimodal retriever based on BridgeTower with cosine similarity improves average score by approximately 3.8% relative to zero-shot and single-modality retrieval; placing vision first within each demonstration ($(\hat I,\hat Q)$4) yields approximately $(\hat I,\hat Q)$5–$(\hat I,\hat Q)$6 point gains over text-first variants; and a brief introductory instruction $(\hat I,\hat Q)$7 adds a stable $(\hat I,\hat Q)$8 on average (Qin et al., 2024). By contrast, switching between random, similar-first, and similar-last inter-demonstration orderings changes performance by less than $(\hat I,\hat Q)$9 point on average (Qin et al., 2024).

Other work elevates “task mapping” from an interpretive metaphor to an explicit design principle. In this view, each demonstration $\hat R$0 defines a local mapping $\hat R$1, while the full context induces a global mapping $\hat R$2 (Li et al., 21 May 2025). Sequence quality can then be assessed by how well these local mappings align with one another and with the query. TACO operationalizes this idea with a lightweight transformer decoder and task-aware attention, and introduces two sequence-level metrics: the Disruption Gap $\hat R$3, measuring sensitivity to replacing one demonstration with a nearest neighbor, and Order Sensitivity $\hat R$4, the standard deviation of accuracy over permutations of the same demonstration set (Li et al., 21 May 2025). On nine datasets and five LVLMs, TACO raises 4-shot VQAv2 average accuracy from 64.13% to 66.75%, OK-VQA from 58.33% to 61.54%, and MSCOCO captioning CIDEr from 118.27 to 119.47 (Li et al., 21 May 2025).

SabER develops a closely related perspective with a decoder-only transformer that autoregressively selects and arranges in-context demonstrations from a library, using task-aware attention and a task guider embedding to refine task mapping hierarchically (Li, 5 Mar 2025). Across five LVLMs and nine benchmarks, its reported improvements include VQAv2 from 57.86 to 64.74, VizWiz from 41.94 to 50.77, and OK-VQA from 49.89 to 57.77 in the 4-shot setting (Li, 5 Mar 2025). These models differ in detail, but both treat demonstration configuration as a learned sequence-construction problem rather than a static similarity search.

Reinforcement learning has also been applied to the same problem. EE-ICL formulates demonstration selection as a sequential decision process over multimodal candidates and learns a policy by combining stochastic beam search with policy-gradient updates (Chen et al., 11 Jun 2025). On four VQA datasets with $\hat R$5 demonstrations, it reports VQAScore improvements over similarity-based retrieval, including $\hat R$6 on OKVQA and $\hat R$7 on VizWiz, with an overall average gain of $\hat R$8 (Chen et al., 11 Jun 2025). These results support a broader conclusion already implicit in task-mapping work: individual demonstrations cannot be evaluated independently, because their joint utility depends on redundancy, complementarity, and compatibility with the query.

5. Internal interventions, compression, and virtual-context methods

Because prompt engineering alone leaves the model’s internal mechanism untouched, a later wave of work intervenes directly in attention, residual streams, or virtual context representations. The resulting methods differ in whether they are training-free, parameter-efficient, or full learned modules, but they share a common objective: to preserve or approximate the effect of demonstrations without relying exclusively on raw concatenation.

Method	Core mechanism	Reported effect
CAMA (Li et al., 21 May 2025)	Training-free modulation of attention logits with a query-ICD joint affinity score $\hat R$9 and positional context factor $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$0	Average gain of $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$1 over standard attention across four LVLMs and six benchmarks
M2IV (Li et al., 6 Apr 2025)	Layer-wise learnable in-context vectors injected into MHA and MLP branches, stored in VLibrary	Average accuracy gain of $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$2 over ICL with the same shot count
HiFICL (Li et al., 13 Mar 2026)	Virtual key-value pairs with low-rank factorization, trained end-to-end as context-aware PEFT	On Idefics2-8B-base: 72.08% VQAv2, 59.56% OK-VQA, 1.2951 COCO CIDEr
AIM (Gao et al., 2024)	Aggregates image information into text-side fused virtual tokens for frozen MLLMs	Drops visual-token burden to a token-ratio of approximately 8% for QWen-VL and 4% for LLaVA-Next
MTV (Huang et al., 2024)	Compresses many-shot demonstrations into attention-head activations and head locations	Encodes 400 examples into $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$3 attention heads; Qwen-VL reaches 45.6% on VizWiz versus 44.3% for 8-shot ICL

CAMA is representative of inference-stage attention recalibration. It starts from a theoretical analysis of multimodal attentional dynamics and identifies three limitations of standard attention: inadequate intra-ICD cross-modal alignment, imprecise query-sample-guided attention, and neglected inter-ICD influences (Li et al., 21 May 2025). Its intervention adds two log-bias terms to the raw attention logits,

$$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$4

where $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$5 is a normalized query-ICD affinity derived from CLIP-based fused embeddings and $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$6 downweights positional bias via $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$7 (Li et al., 21 May 2025). The method is explicitly training-free and plug-and-play.

A second family replaces explicit demonstrations with compact learned surrogates. M2IV learns layer-wise vectors $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$8 that emulate the residual contribution of demonstrations at the MHA and MLP branches, using a self-distillation objective with mimicry, synergy, and supervised losses (Li et al., 6 Apr 2025). AIM instead reads each image-text demonstration with a frozen MLLM, extracts hidden states on the text positions, projects them into fused virtual tokens, and reuses those tokens as text-like substitutes for the original multimodal demonstrations (Gao et al., 2024). HiFICL makes a stronger theoretical claim: the influence of demonstrations in self-attention can be decomposed exactly into a query-dependent mixture of ordinary self-attention and context values, motivating virtual low-rank key-value pairs as a high-fidelity approximation rather than a heuristic shift vector (Li et al., 13 Mar 2026).

A third family targets many-shot scaling and unified multimodal regimes. MTV extracts Multimodal Task Vectors by averaging hidden activations over many batches and identifying attention-head locations where these activations should be injected during inference, thereby encoding many examples into a fixed set of head activations with no extra context length at test time (Huang et al., 2024). UniICL addresses a related problem at the level of a unified model spanning both understanding and generation: its Context-Adaptive Prototype Modulator decouples demonstration encoding, builds an inter-demo prototype bank, and injects adaptive gates into backbone layers, yielding understanding peak 80.9 versus BAGEL’s 72.7 and ICL efficiency $$\hat R \gets \mathcal{M}\{Inst;\,D_1,\dots,D_N;\,(\hat I,\hat Q)\}$$9 on UniICL-Bench (Xu et al., 25 Mar 2026).

Analysis-driven methods also attempt to repair specific internal failures exposed by interpretability work. Mapping-Guided Inference estimates a peak grounding layer $S=\{(x_i,y_i)\}_{i=1}^k$0 by minimizing label-to-image attention entropy across demonstrations, treats the resulting attention pattern as an estimated task mapping $S=\{(x_i,y_i)\}_{i=1}^k$1, and then amplifies the query’s attention to salient demonstration image tokens in later layers (Wang et al., 15 Apr 2026). The gains are deliberately modest—such as 69.09% to 70.17% on Outlier Detection and 48.13% to 48.17% on OK-VQA—but consistent across settings (Wang et al., 15 Apr 2026). This modesty is itself informative: it suggests that inference-time steering can help, yet deeper architectural or training changes may be required to close the multimodal ICL gap.

6. Evaluation regimes, misconceptions, and open directions

A recurring criticism of early evaluation is that conventional VQA and captioning benchmarks do not isolate the capacities that make in-context learning distinctive. VL-ICL Bench was introduced precisely to expand coverage, with eight tasks spanning fast concept binding, fine-grained perception, rule induction, interleaving, long-context learning, and latent-variable induction in generation (Zong et al., 2024). Its results show that even strong models struggle on perception-heavy or context-heavy tasks: in the 2-shot case, GPT-4V reaches 48% on Fast Open MiniImageNet, 30% on CLEVR Count Induction, 84% on Operator Induction, and 50% on TextOCR, while interleaved or long-context tasks remain harder (Zong et al., 2024). This broadening of evaluation reframes multimodal ICL as more than few-shot VQA.

Unified evaluation pushes the taxonomy further. UniICL organizes demonstration roles into six levels—Perception, Imitation, Conception, Deduction, Analogy, and Discernment—and builds UniICL-760K with 766,868 eight-shot episodes across 15 subtasks, plus a 1,250-episode UniICL-Bench covering $S=\{(x_i,y_i)\}_{i=1}^k$2 (Xu et al., 25 Mar 2026). This taxonomy is consequential because it makes explicit that demonstrations do qualitatively different kinds of work across tasks. A plausible implication is that “multimodal ICL” is not a single capability but a family of adaptation behaviors whose failure modes differ by cognitive demand.

One common misconception is that positive shot scaling proves genuine multimodal learning from demonstrations. Multiple studies dispute that inference. Interleaved-data training improves shot scaling but often leaves attention predominantly text-centric (Santos et al., 28 Oct 2025), while broader analyses argue that multimodal ICL frequently reduces to “text-only ICL + superficial image token” whenever text is available (Baldassini et al., 2024). Another misconception is that better retrieval alone solves the problem. ContextNav, an agentic framework for retrieval, denoising, structural alignment, and graph-based workflow orchestration, reports an average ICL gain of 16.8% across six MLLMs versus 7.6% for the best prior baseline, but its own design centers on noise-robust contextualization rather than simple nearest-neighbor search (Fu et al., 6 Oct 2025). The result indicates that scalable retrieval must be coupled with curation and feedback.

Recent work increasingly frames the next stage as a reasoning problem. MMInduction defines an “inductive gap” $S=\{(x_i,y_i)\}_{i=1}^k$3 between answer accuracy and induction accuracy, arguing that models often produce correct answers from flawed reasoning while failing to extract the underlying rule from demonstrations (Wang et al., 4 May 2026). Its response combines similarity-based visual token compression, dynamic attention rebalancing, and an explicit inductive-deductive chain-of-thought, followed by supervised fine-tuning and reinforcement learning with verifiable rewards (Wang et al., 4 May 2026). In a different vein, synthetic-mechanistic analysis shows that multimodal ICL can emerge from induction-style circuits analogous to those studied in text-only transformers, with “previous-token” and “induction” heads remaining causally important across modalities; the same study reports that Rotary Position Embeddings increase the data complexity threshold for ICL (Huang et al., 28 Jan 2026). Taken together, these lines of work suggest that future progress will likely depend on jointly addressing representation alignment, context management, and circuit-level transfer.

The field therefore stands at a transition point. Early progress established that large multimodal models can, under some conditions, adapt from examples in context. Subsequent work clarified that the behavior is often brittle, text-dominated, and highly contingent on sequence design. Current research is moving toward a more mechanistic and systems-level conception in which multimodal ICL is shaped by task mapping, cross-modal alignment, internal attention routing, virtualized context representations, and increasingly diagnostic evaluation suites. Whether future models will turn these ingredients into robust cross-modal induction remains an open question, but the literature already converges on one negative conclusion: raw interleaving of images and text is not, by itself, sufficient.