When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL

Abstract: Implicit multimodal in-context learning compresses demonstrations into internal interventions, ranging from static task vectors to query-conditioned transformations and attention routing. Despite their common goal, these methods differ substantially in how the intervention depends on the query and where it modifies the model, leaving unclear which additional complexity is necessary for a given task. We propose the Selection--Realization Hypothesis. It views demonstrations as inducing a compact family of internal changes from which the query selects, while the model's computation constrains how the selected change can be implemented. We evaluate this account using controlled multimodal tasks in which query dependence varies without changing the underlying task primitives or prompt format. By contrasting correct demonstrations with matched counterfactuals, we measure the structure of explicit M-ICL and test whether it predicts intervention behavior. We find that the success of a static task vector is closely tied to how much of the demonstration-induced change is shared across queries. Additional intervention complexity becomes useful when explicit M-ICL contains query-specific or distributed structure that a local additive shift cannot recover. These relationships extend to natural VQA benchmarks and support cost-aware method selection without access to test performance. Our results provide a unified empirical theory of when demonstrations can be compressed into a task vector and when a more expressive intervention is warranted.

• The paper introduces a diagnostic framework that predicts the minimal sufficient implicit intervention from explicit M-ICL computation, achieving 63.52% average natural-task accuracy with just 0.29 points of mean regret versus the post-hoc best method.
• Static task vectors work best when demonstration-induced changes are shared across queries, while conditional selection helps when query-varying coefficients are predictable, with sharedness correlating 0.96 with static recovery and predictor quality correlating 0.80 with conditional gains.
• The results show that multi-site interventions help when causal support is dispersed and attention routing helps when additive interventions show a fit–recovery gap, enabling 19% lower relative cost than always using distributed vectors and 42% lower cost than always routing.

Motivation and problem statement

Multimodal LLMs (LVLMs) adapt to new tasks from image–text demonstrations via multimodal in-context learning (M-ICL), but explicit demonstrations are expensive: they consume many tokens and must be re-encoded for every query. Implicit M-ICL compresses demonstrations into reusable internal interventions, yet existing methods—static multimodal task vectors (Huang et al., 2024), distributed vectors such as LIVE and M$^2$IV (Peng et al., 2024, Li et al., 6 Apr 2025), query-conditioned shifts such as MimIC (Jiang et al., 11 Apr 2025), and attention routing such as ICR (Li et al., 26 Sep 2025)—are typically evaluated as complete systems. As a result, gains from additional intervention capacity, optimization, and supervision are entangled with the mechanism being tested, and no diagnostic indicates when a static task vector is actually sufficient.

The paper addresses this gap by asking whether the minimal adequate intervention can be predicted from the computation performed by explicit M-ICL itself. An intervention is defined as sufficient when it recovers a fixed fraction of the behavioral gain of explicit M-ICL; among sufficient alternatives, the lowest-cost one is minimal. This reframes method selection as a prediction problem over measurable properties of demonstration-induced computation rather than a post-hoc justification by benchmark accuracy.

The Selection–Realization Hypothesis

The central proposal decomposes implicit M-ICL into two axes. Selection complexity concerns whether the same transformation induced by a demonstration set $D$ is reused across queries or whether its composition varies with the input. Formally, the demonstration-induced change at site $s$, measured against matched counterfactual demonstrations that preserve prompt structure while disrupting the demonstrated mapping, is modeled as a shared component plus a low-rank query-dependent subspace:

$\Delta z_s(x;D) \approx v_s(D) + U_s(D)\,c_s(x,D).$

Realization complexity concerns where and how the selected transformation acts: a localized additive shift at one activation site, coordinated changes across multiple MHA/MLP sites, or modification of attention logits. Static task vectors, multi-site vectors, conditional shifts, and routing become nested cases of a common hierarchy rather than unrelated architectural choices.

Four testable hypotheses follow. H1 states that demonstration-induced changes are compact across queries. H2 states that static sufficiency is governed by the sharedness of the induced energy across queries, and that conditional selection helps only when the remaining coefficients are predictable from the zero-shot query representation. H3 states that when causal support is dispersed across computational sites, multi-site interventions outperform localized ones at matched rank and norm. H4 states that an additive fit–recovery gap—where additive interventions reconstruct explicit-M-ICL representations without recovering behavior—predicts when attention-logit routing is warranted.

Empirical framework

The evaluation uses a controlled task family in which episodes sample rules from a fixed library of visual and textual primitives. Each query carries a gating attribute determining whether it reuses a single rule or selects among episode-specific rules, with conditionality level $\alpha \in [0,1]$ controlling the proportion of rule-selecting queries. An episode-specific permutation maps semantic answers to arbitrary output symbols, forcing both rule inference and output-mapping learning from demonstrations. Five counterfactual demonstration sets per episode apply derangements to output symbols, preserving token lengths, ordering, and marginal label distributions so that only the demonstrated correspondence is disrupted.

Diagnostics are estimated from paired forward passes under correct and counterfactual demonstrations at residual, MHA/MLP output, and attention-logit sites, with queries partitioned into basis-estimation, predictor-training, validation, and test splits. Interventions are compared factorially: static versus oracle versus predicted conditional selection crossed with local, multi-site, and routing realization, all matched in total rank, injected norm, trainable parameters, and supervision. A theory-guided selector then uses the four diagnostics to choose the cheapest admissible family predicted to reach a recovery threshold $\rho = 0.65$, calibrated entirely on controlled data before natural-task evaluation.

Results on controlled tasks

Compactness (H1) holds throughout. Effective rank rises with $\alpha$ (1.1 at $\alpha=0$, 12.4 at $\alpha=1$) and reaches 18.5–28.7 on natural tasks, remaining far below model width. Learned bases achieve held-out reconstruction $R^2$ of 0.55–0.88 versus roughly 0.01–0.05 for random bases, and oracle recovery of 0.63–0.95 versus near zero for random subspaces. Compactness therefore does not imply a single shared direction—the distinction the framework is designed to capture.

Selection (H2) shows the sharpest quantitative pattern. Static additive recovery falls monotonically from 0.92 at $D$0 to 0.33 at $D$1, while oracle conditional conditioning stays between 0.86 and 0.95. Predicted conditioning recovers much of this advantage (average 0.77), and shuffling coefficients across queries removes it (0.51 average), confirming that the gain depends on correct query–transformation correspondence rather than rank or magnitude. Critically, these relationships hold against the measured diagnostics rather than the generating parameter: sharedness predicts static recovery with Spearman correlation 0.96, and held-out coefficient-predictor $D$2 predicts the conditional gain with correlation 0.80. The persistent oracle–predicted gap demonstrates that low sharedness alone does not guarantee that conditioning helps—the coefficients must also be predictable from the query.

Realization (H3–H4) shows that expressivity is not uniformly beneficial. At matched rank and norm, distributing the same transformation across sites improves recovery by only 0.04–0.09, and routing provides little benefit on controlled tasks or VQAv2. However, support dispersion predicts multi-site gain after controlling for strongest-local recovery ($D$3), and the additive fit–recovery gap predicts routing gain after controlling for strongest-additive recovery ($D$4). Routing exceeds the strongest additive intervention by 0.04–0.06 specifically on GQA, OK-VQA, and CVQA—the settings where additive realization fails to convert representational fit into behavioral recovery.

Natural-task validation

Under a frozen 16-shot protocol on VQAv2, GQA, OK-VQA, and CVQA across OpenFlamingo-v2-9B, Idefics2-8B, and LLaVA-NeXT-7B, the theory-guided selector achieves 63.52% average accuracy, within 0.21–0.37 percentage points of the post-hoc best method (mean regret 0.29), despite never observing natural-task test performance. Its relative cost of 0.58 represents a 19% reduction over always deploying M$D$5IV and a 42% reduction over always deploying routing. The diagnostics thus transfer beyond the controlled setting and support cost-aware deployment decisions made before evaluation.

Limitations and open questions

Several constraints qualify these findings. The controlled task family, while varying selection complexity cleanly, uses synthetic scenes with single-token outputs; whether the diagnostics remain predictive for free-form generation or tasks requiring compositional reasoning is untested. The coefficient predictor is a small MLP trained on calibration traces, and the oracle–predicted gap at high $D$6 (0.86 versus 0.64 recovery) indicates that predictability of query-varying coefficients, not merely their existence, remains the binding constraint—a property whose dependence on model scale or training data is not characterized. The sufficiency threshold $D$7 and all diagnostic thresholds are calibrated on controlled episodes; their sensitivity to distribution shift between calibration and deployment conditions is not quantified. Finally, the paper leaves open whether the four diagnostics can be estimated cheaply enough at scale, since the current procedure requires paired traces over hundreds of calibration queries per demonstration set.

Conclusion

This paper provides an empirical theory that connects measurable properties of explicit M-ICL computation to the success of implicit interventions. Static task vectors suffice when demonstration-induced changes are largely shared across queries; conditional selection helps when query-varying coefficients are predictable; multi-site interventions address dispersed causal support; and routing is warranted precisely when additive interventions exhibit a fit–recovery gap. Validated through matched-capacity comparisons, causal controls, and natural VQA benchmarks, the framework replaces post-hoc benchmark justification with evidence-based selection of the minimal sufficient intervention, achieving near-optimal accuracy at substantially reduced cost.