miniReranker: Efficient Multimodal Reranking through Visual Cache Reuse and Interaction Sparsity

Abstract: Multimodal LLMs (MLLMs) have recently shown strong potential as point-wise rerankers by directly modeling query--document relevance through next-token prediction. However, point-wise reranking suffers from substantial repeated computation across query--document pairs, while the causal structure of transformers allows only prefix segments to be reused via pre-caching. To address the misalignment of existing query-first and document-first formats with both VQA-style prompting and computation-aware reuse, we propose a \textit{vision-first} formulation that improves both cache reuse efficiency and reranking performance. However, the remaining cost is still considerable and stems from three main sources: (1) \textit{model depth}, for which we reduce active parameters via early exit; (2) \textit{cross-segment attention}, which we restrict to a narrow interaction band across a few layers; and (3) \textit{visual tokens}, where we reduce the number of tokens via embedder-guided pruning. Together, these designs form miniReranker, which reduces reranking runtime to <1% of the dense implementation under high-reuse settings for a single query, while preserving >96% of the dense model performance.

• The paper introduces miniReranker, which dramatically reduces computational cost in multimodal reranking by reusing visual caches and reordering prompt inputs.
• It employs architectural optimizations such as early exit, interaction bands, and embedder-guided token pruning to maintain over 96% of dense model performance.
• Experimental results on Qwen3-VL models show significant latency reductions for image, video, and document tasks with minimal accuracy loss.

miniReranker: Efficient Multimodal Reranking through Visual Cache Reuse and Interaction Sparsity

Introduction

Multimodal LLMs (MLLMs) have been increasingly adopted as point-wise rerankers in retrieval pipelines for vision, language, and document tasks. While these models introduce fine-grained token-level cross-modal interactions that significantly enhance relevance estimation over dual-encoder architectures, the computational cost per query-document pair is substantial due to repeated processing of redundant multimodal information and inefficiencies in model operations. The "miniReranker" framework addresses these challenges by proposing a suite of architectural and inference-time optimizations focused on maximizing computational reuse and eliminating redundancy at multiple levels, specifically targeting the expensive visual modality and transformer depth.

Vision-First Prompt Reformulation

Standard MLLM rerankers rely on the query-first sequence order inherited from textual retrieval, often breaking the model’s alignment with pre-training objectives, especially in image or video retrieval tasks. The "vision-first" reformulation proposed by miniReranker dynamically orders sequence inputs such that the visual content always precedes the text, restoring alignment with the VQA-style pre-training and maximizing the potential for prefix key-value (KV) cache reuse. Specifically:

• In text-to-vision (T→V) settings, the document (visual) is placed before the query, enabling offline visual feature caching and significant online savings.
• In vision-to-text (V→T), the visual query is placed first, leveraging computation amortization across reranked documents.
• In vision-to-vision (V→V), the document precedes the query, supporting large-scale corpus caching.

This dynamic ordering yields order-of-magnitude reductions in online floating-point operations (FLOPs) compared to rigid baselines, particularly as the length and volume of visual sequences scale.

Figure 1: Overview of miniReranker architecture, showcasing vision-first reformulation and three core compression strategies: Early Exit, Interaction Bands, and Token Pruning.

Model-Level Compression: Depth, Attention, and Token Sparsification

Despite prompting optimization, vanilla MLLM rerankers retain depth-wise, attention-wise, and token-wise redundancy. miniReranker introduces three orthogonal strategies:

• Early Exit: Layer-wise logit probing reveals that reranking decisions saturate at intermediate layers, with ≈95% of final-layer performance achieved by 60% of the transformer depth. Forward computation is thus safely truncated at predefined early-exit layers without significant accuracy loss.

  Figure 2: Layer-wise logit probing exposes depth-wise redundancy and localizes effective cross-segment interaction to a narrow band of transformer layers.

• Interaction Bands: Cross-segment (query-document) attention is restricted exclusively to a range of intermediate layers identified by logit probing and cosine similarity analysis. By masking cross-segment attention elsewhere, computation becomes sparse and memory demands are substantially reduced.
• Embedder-Guided Token Pruning: Visual tokens are pruned based on aggregated attention scores harvested from the first-stage retrieval embedder, incurring zero additional computation. Selecting the top $\alpha\%$ of tokens achieves aggressive visual compression—with 50% compression typically retaining near-dense baseline accuracy.

Each component provides complementary efficiency; ablations confirm that combining all three reductions with vision-first prompting yields the greatest latency and parameter savings while minimally impacting effectiveness.

Experimental Results and Numerical Highlights

miniReranker is instantiated on Qwen3-VL across 2B, 4B, and 8B parameter scales. All experiments are conducted on the MMEB-v2 benchmark, spanning 78 tasks across images, visual documents, and video.

• Effectiveness: miniReranker preserves >96% of reranking performance (e.g., for the 2B model, 70.3 overall vs. 73.0 for dense) on MMEB-v2. For the 4B/8B models, >98% of the full performance is maintained.
• Efficiency:
  • Active parameters are reduced to 58% of the dense baseline via early exit.
  • Training speed accelerates by nearly 3×.
  • Online reranking latency is drastically reduced:
  • For video reranking, runtime is reduced to <1% of dense.
  • For image reranking, <15% of original latency is achieved.
  • Latency savings scale with corpus/query size due to cache reuse.
  • 

  • Figure 3: Latency scaling in the vision-as-query setting – miniReranker achieves a significant reduction in latency as the candidate pool grows.

  • 

  • Figure 4: Latency scaling in the vision-as-document setting, demonstrating effective reuse as query volume increases.

  • 

  • Figure 5: Latency breakdown of individual compression components across modalities – early exit dominates gains in image tasks, token pruning and interaction bands are critical for long-sequence video.

• Prompt Optimization: Vision-first formulations outperform query/document-first even after fine-tuning, indicating generalizability of the prompt strategy.

Detailed Analysis: Redundancy and Compression Dynamics

Layer-wise probing and similarity analyses confirm that, unlike open-ended VQA or generation tasks that require deep transformer reasoning, binary yes/no reranking decisions emerge after few cross-modal exchange layers. Probing on general VQA tasks shows that such tasks require much deeper layers compared to point-wise reranking, justifying the specificity of early exit for relevance estimation.

Visual token ablations reveal that leveraging retrieval-step attention as a token-importance prior is preferable to alternatives based on similarity or internal vision-encoder attention. This ensures semantic evidence necessary for discrimination is preserved while aggressively trimming computational paths.

Practical and Theoretical Implications

miniReranker demonstrates that high-quality multimodal relevance estimation is compatible with orders-of-magnitude reduction in inference and training cost through architectural and runtime optimizations that explicitly target modality and structural redundancies. This has direct implications for large-scale search and retrieval-augmented generation systems with multimodal corpora. Practically, it enables deployment in resource-constrained or latency-sensitive environments. Theoretically, these findings further support the modularity of signal emergence in transformers and establish a principled framework for the design of specialized, efficient MLLM subroutines distinct from generative tasks.

Future research directions include extending the proposed compression strategies beyond point-wise reranking to richer list-wise or agentic retrieval paradigms, as well as generalizing attention-localization and early exit to other structured MLLM applications.

Conclusion

miniReranker introduces a comprehensive approach to compressing and accelerating point-wise multimodal reranking by exploiting visual cache reuse, transformer depth redundancy, localized cross-modal attention, and semantic token sparsity. The framework approaches dense-model effectiveness with a fraction of the computational cost, providing a robust and scalable solution for modern multimodal retrieval systems.

The clear performance-efficiency trade-off illustrated in these results signals a promising direction for future MLLM research and deployment.

Figure 6: Layer-wise probing on general VQA tasks – final-layer performance is only recovered at much deeper layers, highlighting the specificity of early exit for point-wise reranking.

Figure 7: End-to-end latency including visual pre-encoding and cache construction overhead in the vision-as-query setting.

Figure 8: End-to-end latency including visual pre-encoding and cache construction overhead in the vision-as-document setting.