RetLLM: Multimodal Retrieval Framework

• RetLLM is a multimodal retrieval framework that generates similarity scores using pretrained MLLMs instead of relying on contrastive embedding learning.
• It employs a coarse-to-fine pipeline that combines CLIP-based top‑k candidate filtering with a fine-stage MLLM scorer enhanced by visual re-injection and entropy-based tie-breaking.
• The framework achieves competitive zero-shot performance across benchmarks without fine-tuning or labeled data, balancing retrieval accuracy and computational efficiency.

RetLLM is a framework for multimodal information retrieval (MMIR) that treats retrieval as a similarity score generation problem for multimodal LLMs (MLLMs), rather than as contrastive embedding learning. In its main formulation, a query and candidate can each be text, image, or interleaved image-text content, and the system operates in a training- and data-free manner: no finetuning, no contrastive training, no reranker training, and no use of labeled retrieval pairs or task-specific supervision for adaptation. Its core pipeline combines a coarse top-$k$ filtering stage with a fine-stage MLLM scorer, and adds visual enhancement and entropy-based tie-breaking to improve retrieval quality (Su et al., 25 Feb 2026).

1. Conceptual framing and problem definition

RetLLM addresses MMIR in a setting where a query $q$ is matched against a candidate set $\Omega=\{c_1,\dots,c_N\}$, with the additional complication that both queries and candidates may be unimodal or multimodal. This generalizes standard image-text retrieval to cases involving long-form text, mixed-modal queries, and compositional retrieval. The framework is motivated by two limitations of recent MLLM-based retrieval systems: dependence on large labeled multimodal datasets, and pre-training inconsistency caused by contrastive finetuning objectives that differ from autoregressive multimodal reasoning objectives (Su et al., 25 Feb 2026).

The distinctive claim of RetLLM is that retrieval need not be reduced to embedding alignment. Instead, the MLLM itself is prompted to act as a similarity score generator. In this view, retrieval becomes conditional multimodal reasoning over a query-candidate pair. This preserves the native inference mode of an MLLM—token generation conditioned on multimodal context—rather than converting the model into a universal encoder optimized by a contrastive loss.

The fine-stage retrieval score is written as

$$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$

Here $f_i$ is the generated retrieval score for candidate $c_i$ inside a reduced candidate pool $\mathcal{C}$. A plausible implication is that RetLLM is architecturally closer to an MLLM reranker than to a classic dense retriever, even though its first stage still uses an embedding model for candidate filtering.

2. Coarse-to-fine retrieval pipeline

RetLLM uses a coarse-then-fine framework to reduce the cost of direct MLLM scoring. The coarse stage builds a small candidate pool by applying cosine similarity in embedding space: $$\mathcal{C} = \mathrm{TopK}(s), \quad s_i=\frac{\mathbf{q}^\top \mathbf{c}_i}{\|\mathbf{q}\| \|\mathbf{c}_i\|}, \quad i=1,2,\dots,N.$$ In the main experiments, the coarse retriever is CLIP-ViT-L/14@336px and the default pool size is top-5 (Su et al., 25 Feb 2026).

This stage is explicitly justified by scalability and by hard-sample focus. Scoring every candidate with an MLLM would require $N$ multimodal forward passes per query. The coarse stage reduces this to $K$ MLLM calls, where only semantically plausible candidates remain. The paper presents this as a practical compromise between the speed of embedding retrieval and the reasoning precision of direct MLLM-based scoring.

The fine stage then jointly feeds the query and each retained candidate into an MLLM through a multimodal instruction template that asks the model to predict a semantic similarity score. Final selection is described as ranking candidates by those scores. Although the ranking rule is not printed as a standalone equation in the paper, it is directly implied by the method description: $q$0

The framework also includes an entropy-based decision rule for ties. When multiple candidates receive the same top score, RetLLM issues a second prompt of the form “<query>, <candidate>. Does the candidate match the query, True or False.” It then computes

$q$1

where $q$2 is vocabulary size and $q$3 is the softmax probability at the last token position, and selects

$q$4

Lower entropy is interpreted as greater model confidence. This tie-breaking mechanism indicates that score generation alone can be ambiguous, especially when multiple semantically close candidates survive coarse filtering.

3. Visual enhancement and inference-time reasoning support

A central auxiliary component is the visual enhancement module, introduced to help the MLLM “re-pick forgotten visuals” during fine-stage reasoning. The motivation is that multimodal generation can over-rely on linguistic priors and underuse fine-grained visual evidence. This is particularly problematic in retrieval, where top candidates are often differentiated by subtle local attributes rather than broad semantics (Su et al., 25 Feb 2026).

The module is formulated as inference-time visual re-injection inside the feed-forward network (FFN) of Transformer blocks. Starting from the standard FFN

$q$5

the paper rewrites it as a key-value memory: $q$6 so that

$q$7

Visual tokens $q$8 are then treated as supplementary key-value entries. The correction term is

$q$9

and the enhanced FFN output is

$\Omega=\{c_1,\dots,c_N\}$0

with $\Omega=\{c_1,\dots,c_N\}$1. In experiments, $\Omega=\{c_1,\dots,c_N\}$2 (Su et al., 25 Feb 2026).

This mechanism is neither iterative chain-of-thought prompting nor a separate visual encoder training stage. It is an architectural inference-time augmentation intended to restore underutilized visual evidence inside the MLLM’s internal computation. The paper reports that removing visual enhancement lowers performance, especially on COCO image-to-text retrieval, which suggests that visual fidelity remains a limiting factor even when an MLLM is directly prompted to score relevance.

4. Empirical performance and benchmark coverage

RetLLM is evaluated zero-shot on six named benchmarks: Flickr30K, COCO, ShareGPT4V, Urban1K, SugarCrepe, and MMEB. The tasks include image-to-text retrieval, text-to-image retrieval, and compositional retrieval. The primary metric is Recall@1, while MMEB uses average Precision@1 over datasets and meta-tasks (Su et al., 25 Feb 2026).

The main setting uses Qwen2.5-VL-7B as the MLLM, CLIP-ViT-L/14@336px as the coarse retriever, top-$\Omega=\{c_1,\dots,c_N\}$3, and $\Omega=\{c_1,\dots,c_N\}$4 for visual enhancement. The reported results show strong zero-shot performance across short-caption, long-caption, and compositional retrieval benchmarks.

Benchmark / Task	Setting	Result
Flickr30K	$\Omega=\{c_1,\dots,c_N\}$5	94.5
Flickr30K	$\Omega=\{c_1,\dots,c_N\}$6	82.0
COCO	$\Omega=\{c_1,\dots,c_N\}$7	70.4
COCO	$\Omega=\{c_1,\dots,c_N\}$8	54.1
ShareGPT4V	$\Omega=\{c_1,\dots,c_N\}$9	97.6
ShareGPT4V	$$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$0	94.2
Urban1K	$$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$1	88.9
Urban1K	$$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$2	78.6
SugarCrepe	Replace	94.8
SugarCrepe	Swap	92.7
SugarCrepe	Add	96.2

On MMEB, RetLLM reaches 60.3 on Classification, 27.8 on VQA, 62.4 on Retrieval, 60.2 on Grounding, 52.0 on IND, 50.2 on OOD, and 54.2 overall (Su et al., 25 Feb 2026).

The paper emphasizes that RetLLM often outperforms zero-shot embedding baselines and is competitive with or better than training-based MLLM retrievers on many tasks. At the same time, it does not dominate every benchmark. Urban1K is the clearest stated exception, where UniME scores higher than RetLLM. This matters because the paper’s claim is not universal superiority, but broad competitiveness without any retrieval-specific training.

5. Ablations, scaling behavior, and operational trade-offs

The ablation results indicate that each component contributes measurable gains. The paper compares the full system against variants with entropy only, enhancement only, and MLLM-only scoring. Reported values include the following (Su et al., 25 Feb 2026):

Setting	Flickr30K $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$3	Flickr30K $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$4	COCO $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$5	COCO $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$6
ALL	94.5	81.8	69.2	52.1
entropy only	94.0	81.2	68.7	50.8
enhancement only	94.0	80.7	68.8	51.5
MLLM only	93.6	80.2	66.9	50.3

These ablations support three claims. First, the full system consistently outperforms direct MLLM scoring. Second, visual enhancement is especially relevant for preserving visual fidelity. Third, entropy-based selection helps when score ambiguity remains after fine-stage reasoning.

The top-$$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$7 study uses $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$8. The paper’s stated conclusion is that larger $$f_i = \mathrm{MLLM}(q, c_i), \quad c_i \in \mathcal{C}.$$9 improves recall but increases computational cost, and that $f_i$0 is the best practical balance. This establishes the central trade-off in RetLLM’s design: it is more scalable than brute-force MLLM retrieval, but still more expensive than pure embedding retrieval.

RetLLM also scales with stronger foundation models. Using stronger CLIP backbones improves candidate filtering, particularly on long-text datasets. On ShareGPT4V, ViT-B gives 94.8 / 88.1, ViT-L gives 97.6 / 94.2, and Long-CLIP-L gives 96.6 / 95.1. On Urban1K, ViT-B gives 84.0 / 71.8, ViT-L gives 88.9 / 78.6, and Long-CLIP-L gives 95.2 / 95.8. Stronger MLLMs also help: Phi-3.5-V yields 86.5 / 72.3 on ShareGPT4V, Qwen2-VL yields 93.8 / 94.1, and Qwen2.5-VL yields 97.6 / 94.2 (Su et al., 25 Feb 2026).

Several limitations are explicit. The method depends on coarse-stage recall: if the true item is filtered out of top-$f_i$1, the fine stage cannot recover it. Inference remains costlier than single-pass embedding retrieval. The need for entropy tie-breaking suggests that generated scores can be coarse or unstable. A plausible implication is that RetLLM is best suited to settings where reasoning quality matters enough to justify reranking cost.

6. Position within the broader literature and common confusions

RetLLM is specifically a multimodal retrieval framework based on score generation by pretrained MLLMs. It should be distinguished from similarly named systems that address different problems.

“RET-LLM: Towards a General Read-Write Memory for LLMs” proposes an explicit external memory that stores knowledge as triplets and supports textual MEM_WRITE and MEM_READ operations. Its focus is structured fact storage, retrieval, and updateable temporal knowledge, rather than multimodal candidate ranking (Modarressi et al., 2023). By contrast, RetLLM does not define an explicit read-write fact memory. Its retrieval target is a candidate item from a query-specific pool, and its main computation is pairwise or setwise relevance scoring in a coarse-to-fine pipeline.

RetLLM is also distinct from “Retrv-R1,” a retrieval-specialized reasoning MLLM that performs explicit set-level chain-of-thought reasoning over a top-$f_i$2 candidate set and introduces an information compression module, details inspection, synthetic retrieval CoT activation, and GRPO-based reinforcement learning (Zhu et al., 3 Oct 2025). RetLLM is training-free and data-free; Retrv-R1 is explicitly trained. This difference is methodologically central: RetLLM asks what pretrained MLLMs can already do for retrieval without adaptation, whereas Retrv-R1 studies retrieval-specific reasoning after staged training and RL.

A further source of confusion is “ReaLLM,” which concerns real-time streaming generation and Speech ReaLLM for automatic speech recognition, not retrieval (Seide et al., 2024). The name similarity is incidental rather than conceptual.

Within MMIR research, the main contribution of RetLLM is therefore not a new encoder, a retrieval-specific training objective, or a new external memory abstraction. Its contribution is a reformulation: retrieval is cast as direct multimodal score generation by an off-the-shelf MLLM, made practical by top-$f_i$3 filtering, visual re-injection, and entropy-based decision rules. This suggests that some of the gains usually attributed to specialized retrieval finetuning can instead be obtained by preserving and exploiting pretrained multimodal reasoning behavior (Su et al., 25 Feb 2026).