Parallel In-context Learning for Large Vision Language Models

Abstract: Large vision-LLMs (LVLMs) employ multi-modal in-context learning (MM-ICL) to adapt to new tasks by leveraging demonstration examples. While increasing the number of demonstrations boosts performance, they incur significant inference latency due to the quadratic computational cost of Transformer attention with respect to the context length. To address this trade-off, we propose Parallel In-Context Learning (Parallel-ICL), a plug-and-play inference algorithm. Parallel-ICL partitions the long demonstration context into multiple shorter, manageable chunks. It processes these chunks in parallel and integrates their predictions at the logit level, using a weighted Product-of-Experts (PoE) ensemble to approximate the full-context output. Guided by ensemble learning theory, we introduce principled strategies for Parallel-ICL: (i) clustering-based context chunking to maximize inter-chunk diversity and (ii) similarity-based context compilation to weight predictions by query relevance. Extensive experiments on VQA, image captioning, and classification benchmarks demonstrate that Parallel-ICL achieves performance comparable to full-context MM-ICL, while significantly improving inference speed. Our work offers an effective solution to the accuracy-efficiency trade-off in MM-ICL, enabling dynamic task adaptation with substantially reduced inference overhead.

• The paper introduces Parallel-ICL, which clusters multimodal demonstrations and combines parallel chunk predictions with a weighted Product-of-Experts to reduce long-context latency without training or architectural changes.
• Experiments across multiple large vision-language models show up to 1.78× standalone speedups and up to 3.0× with demonstration pruning, while often matching or exceeding full-context accuracy.
• Ablations show that multimodal k-means clustering and query-relevance weighting improve diversity and accuracy, but selecting the chunk count remains a key limitation because it balances relevance against parallel efficiency.

Motivation: the accuracy–latency trade-off in MM-ICL

Multi-modal in-context learning (MM-ICL) allows large vision-LLMs (LVLMs) to adapt to unseen tasks by conditioning decoding on a set of demonstration (image, question, answer) tuples, without parameter updates. Performance generally improves with the number of demonstrations $N$, but because Transformer attention scales quadratically in context length—and LVLMs consume thousands of visual tokens per image—the latency cost of long demonstration contexts is severe. A preliminary analysis with LLaVA-OV-7B on MI-Bench-ICL makes this concrete: moving from zero-shot to 32-shot MM-ICL raises accuracy from 0.00 to 58.90 but increases total latency from 0.099 s to 3.479 s, and the prefill phase dominates this cost (3.444 s of the 3.479 s at 32 shots). Existing efficiency approaches for MM-ICL, notably task-vector methods such as LIVE (Arpanaei et al., 2024) and Mimic-ICL, require hundreds of demonstrations and per-task optimization before inference, which undermines the dynamic, plug-and-play character that motivates MM-ICL in the first place. The central research question is whether long-context MM-ICL can be approximated at inference time without such additional burdens.

Method: chunking and compiling contexts

The proposed Parallel-ICL exploits the observation that demonstrations are largely independent and need not be serialized into one long context. It operates in two stages. Context chunking partitions the demonstration set $\mathcal{E}$ into $K$ chunks via k-means clustering on multi-modal CLIP features $\psi(x,t)$, concatenating image and text embeddings. Context compilation computes each chunk's next-token logits with parallel batch-wise forward passes and aggregates them as a weighted Product-of-Experts (PoE):

$$\hat{l}_\theta(y_i)=\sum_{k=1}^{K} w_k\, l_\theta(y_i\mid \bm{y}_{<i}, C_k, x, t).$$

The PoE formulation is preferred over a mixture-of-experts for high-dimensional vocabularies, following prior evidence that logit-level PoE ensembling outperforms MoE in LLM ensemble reasoning.

The design choices are grounded in an information-theoretic decomposition of the Fano-style ensemble error bound, which separates the bound into a relevance term $I_\mathrm{relev}$ (correlation between each member's output and the ground truth) and a redundancy term $I_\mathrm{redun}$ (duplicated information across members). This yields two desiderata: maximize inter-chunk diversity and prioritize query-relevant members. Accordingly, clustering-based chunking constructs internally coherent chunks whose predictions are mutually diverse, while compilation weights $w_k$ are set by a softmax over cosine similarities between the query's multi-modal feature and the mean features of each chunk's questions. The authors acknowledge that the exact mutual-information terms are not computable prior to inference; they therefore rely on proxy metrics based on KL divergence between chunk predictive distributions to validate these properties empirically.

Empirical findings

Experiments cover LLaVA-OV-7B, Qwen2.5-VL-7B, and InternVL3.5 (8B/14B/38B) across MI-Bench-ICL (C-VQA, O-VQA, Hallucination, Demo), GQA, TextVQA, and COCO Caption, using up to four H100 GPUs with greedy decoding.

Accuracy can exceed full-context ICL. In the proof-of-concept on MI-Bench-ICL (Demo), Parallel-ICL with $K \in \{2, 4\}$ consistently outperforms full-context MM-ICL at $N=32$ across all three LVLMs—for example, LLaVA-OV-7B reaches 63.40 accuracy versus 58.90 for the full context. The authors attribute this to mitigation of the "lost in the middle" effect [2404.xxxx], where information deep in a long context is under-utilized; shorter, focused chunks appear to make all demonstrations accessible. This is a strong claim: an approximation method surpassing its target suggests the full-context baseline itself is suboptimal at large $\mathcal{E}$0, though the paper does not isolate the mechanism beyond this hypothesis.

Speedups are substantial and composable. At 32 shots, Parallel-ICL alone achieves up to 1.78× speedup with approximation ratios of roughly 96–102% relative to full-context MM-ICL. Combined with DivPrune-based demonstration reduction, the best overall trade-offs emerge—on Qwen2.5-VL-7B, Parallel-ICL + DivPrune attains a 3.0× speedup with a 101.9% approximation ratio, and it achieves 102.2% on InternVL3.5-8B at 2.5× speedup. By contrast, DivPrune alone degrades performance sharply on some models (78.6% approximation ratio at 8 shots on Qwen2.5), indicating that simply reducing demonstrations loses task knowledge that chunk-level ensembling preserves. Notably, the combined configuration even exceeds full-context accuracy on Qwen2.5's Demo partition (72.60 vs. 58.90).

Ablations confirm the principled design. Replacing k-means chunking with random assignment drops diversity from 0.378 to 0.247 ($\mathcal{E}$1) and accuracy from 63.40 to 59.30; replacing similarity-weighted compilation with uniform weights drops relevance from 0.854 to 0.777 and accuracy to 60.00. Multi-modal features outperform text-only or vision-only features for both stages, consistent with prior findings on multi-modal task representations. All ablated variants are slightly faster, so the principled components add negligible overhead.

Scaling behavior holds. Across InternVL3.5 at 8B, 14B, and 38B parameters, Parallel-ICL maintains or improves accuracy (e.g., 87.40 vs. 86.40 for the 38B model at $\mathcal{E}$2) while delivering 1.16×–1.36× speedups, indicating the approach is not sensitive to scale within the tested range.

A recurring empirical pattern is a diversity–relevance trade-off: increasing $\mathcal{E}$3 raises inter-chunk KL divergence but reduces per-chunk relevance, since each chunk contains fewer demonstrations, and larger $\mathcal{E}$4 does not always reduce latency because the input image and query text must be re-encoded per chunk. Selecting $\mathcal{E}$5 is therefore framed as balancing these competing quantities rather than maximizing either.

Limitations and open questions

Two limitations are stated explicitly. First, when $\mathcal{E}$6 is small and $\mathcal{E}$7 is large, the repeated processing of the query image and text across chunks can outweigh the parallelism benefit, producing no net speedup (e.g., $\mathcal{E}$8); the paper manages this only through heuristic selection of $\mathcal{E}$9 given $K$0. Second, $K$1 is a critical hyperparameter governing the diversity–relevance trade-off and is chosen via validation sets; no automatic or adaptive criterion for selecting $K$2 is provided. Additionally, several points remain open: the causal explanation for why chunked ensembling beats the full context at $K$3 (the "lost in the middle" hypothesis is asserted, not verified mechanistically); whether the observed gains extend to interleaved-image or video settings, where demonstration independence may be weaker; and how the proxy KL-based diversity/relevance metrics relate to the true mutual-information quantities in the Fano bound, which were deemed computationally intractable over the LVLM vocabulary.

Conclusion

Parallel-ICL reframes MM-ICL inference as a weighted Product-of-Experts ensemble over clustered, short demonstration contexts, guided by ensemble-learning principles of diversity and relevance. It delivers up to 1.78× speedup standalone and up to 3.0× when composed with token/demonstration pruning, while matching or exceeding full-context accuracy—with the notable finding that chunked contexts can outperform the full context at 32 shots. The method requires no training, task vectors, or architectural changes, and its orthogonality to general LVLM efficiency techniques positions it as a complementary component. The main unresolved issues are adaptive selection of the chunk count, the unverified mechanism behind accuracy gains over full-context ICL, and applicability beyond single-image demonstration formats.