Generative Universal Multimodal Retrieval with Dual-role Identifiers

Abstract: Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifier-based GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. Extensive experiments on the M-BEIR benchmark and the text-to-image evaluation datasets show that:(1)DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers. (2)Ablation and scaling analyses reveal how the base LMM, beam size, reranking depth, and fusion strategy affect retrieval performance, providing practical guidance for system design.

• The paper introduces DrIG, a generative universal multimodal retriever that uses residual-quantized identifiers both as ordered decoding sequences and order-invariant relevance signals, with dense reranking to recover lost semantic detail.
• DrIG raises average M-BEIR performance from 29.5 to 38.0 locally and from 28.6 to 36.4 globally over GENIUS, while achieving strong zero-shot Flickr30K retrieval with Recall@1/5/10 of 59.0/83.1/88.2.
• The results show that contrastive embedding alignment, Trie-constrained decoding, set-based guidance, and query augmentation are central to performance, although identifier compression still trails dense retrievers on knowledge-intensive text retrieval.

Motivation and problem setting

Generative information retrieval (GIR) replaces the conventional index-retrieve-then-rank pipeline with a model that directly generates discrete identifiers of relevant items. While GIR has been studied extensively for unimodal text retrieval and, more recently, for task-specific cross-modal retrieval, its application to universal multimodal retrieval (GUMR)—where queries and candidates may be text, images, or image–text pairs, with intent specified by natural-language instructions—remains underexplored. The paper identifies three obstacles: constrained left-to-right decoding is vulnerable to prefix-level pruning errors and local optima; identifier design must jointly capture modality distinctions and fine-grained semantics; and discretization of continuous embeddings loses information needed for fine-grained ranking, leaving generative retrievers behind strong dense baselines.

The proposed framework, DrIG (2608.12987), addresses these issues with a single design decision: each candidate receives one residual-quantized (RQ) identifier that serves two complementary roles—an ordered sequence for autoregressive generation under a Trie constraint, and an unordered set providing a prefix-independent relevance prior that guides beam search. A hybrid reranking stage over dense embeddings compensates for quantization loss.

Framework

DrIG proceeds in four stages. First, a Qwen2-VL large multimodal model (LMM) encodes queries and candidates into a shared 3584-dimensional embedding space using explicit one-word summarization prompts, followed by LamRA-style two-stage contrastive fine-tuning (text-to-text on NLI data, then multimodal tasks from M-BEIR) with InfoNCE loss. Second, candidate embeddings are converted into identifiers via modality-aware residual quantization: the first codebook has size 3 to encode modality (image, text, image–text pair), and subsequent codebooks encode progressively finer semantics. Identifier construction is trained with three losses—RQ reconstruction, in-batch contrastive alignment of query–candidate pairs, and an MSE loss preserving pairwise consistency after quantization—with code embeddings updated by EMA.

Third, the set-based role is operationalized by mapping the query embedding into the global code-token vocabulary through an MLP and aggregating token-level scores over the identifier's tokens, yielding an order-invariant relevance score $s_{\mathrm{oi}}$. This is optimized with a contrastive objective plus a margin ranking loss against the hardest in-batch negative selected by dense cosine similarity.

Fourth, a T5-small decoder (30M parameters), conditioned on projected query prefix embeddings, generates identifiers via Trie-constrained beam search. The expansion score combines the prefix-validity constraint, the autoregressive decoding score, and a weighted global prior $\lambda\phi$, where $\phi$ takes the maximum $s_{\mathrm{oi}}$ over all candidates sharing the current prefix. Training uses token-level cross-entropy on interpolated augmented queries ($$\mathbf{z}'_q = \mu \mathbf{z}_q + (1-\mu)\mathbf{z}_c$$ with $\mu \sim \mathrm{Beta}(\alpha,\alpha)$) plus a smooth pairwise ranking loss with an adaptive margin derived from teacher cosine similarities. At inference, the top-$k$ generated candidates can be reranked by dense similarity at cost $\mathcal{O}(kd)$ with $k \ll |\mathcal{C}|$.

Main results on M-BEIR

On M-BEIR (10 datasets, 4 domains, 8 task types, 5.6M candidates), evaluated under local-pool and global-pool settings, DrIG improves the average score over GENIUS—the prior state-of-the-art generative GUMR method—from 29.5 to 38.0 locally (+28.8% relative) and from 28.6 to 36.4 globally (+27.3%). Gains are largest on knowledge-intensive and compositional tasks; InfoSeek multimodal-to-text R@5 rises from 11.4 to 25.0 locally (+119.3%) and from 9.9 to 22.2 globally (+124.2%). Notably, the local-to-global degradation is moderate (38.0 → 36.4), comparable to dense baselines such as CLIP-SF (51.3 → 48.6), suggesting robustness to pool expansion attributable to instruction-conditioned decoding and the modality-aware first token.

With dense reranking, DrIG-C and DrIG-LT reach averages of 48.7 and 50.4 (local) and 47.1 and 48.9 (global), outperforming GENIUS-C and BLIP-FF, with DrIG-LT competitive with CLIP-SF globally. However, the paper concedes a substantial residual gap to LamRA (63.7 local, 61.4 global); the most striking example is WebQA text-to-text, where DrIG-LT achieves 65.9 versus LamRA's 96.7. The authors attribute this to information loss when knowledge-dense long texts are compressed into short identifier sequences—a limitation reranking mitigates but does not eliminate.

Text-to-image transfer

On Flickr30K, DrIG trained only on M-BEIR achieves zero-shot Recall@1/5/10 of 59.0/83.1/88.2, already exceeding several in-domain generative baselines (GRACE, IRGen, AVG, ComGTIR-D). With LamRA-based reranking, DrIG-LT reaches 75.8/90.0/91.6 zero-shot, surpassing the strongest prior hybrid baseline ComGTIR-DH$_{\mathrm{clip}}$ (68.4/86.3/90.7); in-domain training further raises this to 76.9/92.5/94.8. On MSCOCO, DrIG-LT reaches 56.1/79.6/86.0 with M-BEIR training, with only marginal gains from in-domain fine-tuning (56.3/80.1/86.7), indicating that universal training provides a strong foundation for standard text-to-image retrieval. Reranking yields larger gains at Recall@1 than Recall@10, implying the generative retriever already surfaces relevant candidates but ranks them imperfectly.

Component ablations

The ablation study isolates each component on representative M-BEIR tasks:

Variant	MSCOCO $\lambda\phi$0 R@1	WebQA $\lambda\phi$1 R@1	CIRR R@5
Full DrIG	41.8	20.6	33.7
w/o Set-based role	40.8	19.7	32.2
w/o Trie	20.6	18.9	14.6
w/o Trie + Set-based	0.1	0.3	0.1
w/o Query augmentation	37.1	14.9	28.8
w/o $\lambda\phi$2	40.3	20.1	31.9
w/o Contrastive loss $\lambda\phi$3	0.5	0.0	0.1
w/o Modality codebook	40.6	19.7	33.4

Two results stand out as near-categorical. Removing the pre-quantization contrastive loss collapses performance to near zero across all tasks (e.g., MSCOCO text-to-image R@1 falls from 41.8 to 0.5), establishing that residual quantization alone cannot produce retrieval-oriented identifiers—the embedding space must first be organized by relevance and intent. t-SNE visualizations corroborate this, showing compact, modality-separated clusters only with the contrastive objective. Second, removing both the Trie constraint and the set-based role reduces most metrics to near zero, while removing either alone degrades performance moderately; valid-prefix control and global relevance guidance are thus complementary rather than redundant. Query augmentation contributes substantially (MSCOCO image-to-text R@1 drops from 57.8 to 43.9 without it), whereas the ranking loss and modality codebook yield smaller, task-dependent gains—the latter slightly hurts NIGHTS image-to-image retrieval (R@5 rises from 17.7 to 19.1 without it), which the authors explain by the single-modality candidate pool offering no benefit from an explicit modality token.

Effectiveness–efficiency trade-off

Throughput measurements on a single A100 show generative methods maintaining nearly flat QPS as the image pool grows from 5K to 300K (GENIUS ~18.4, DrIG ~17.4, DrIG-C ~17.0 QPS), while dense baselines degrade with pool size and LMM-reranked LamRA is slowest. Beam size governs a steep trade-off: QPS falls from 972.6 at beam 1 to 37.6 at beam 50, while MSCOCO text-to-image R@10 climbs from 25.1 to 79.4, saturating around beam 20–30. Codebook depth matters more than width: increasing levels from 4 to 8 at $\lambda\phi$4 lifts WebQA text-to-text R@1 from 7.3 to 20.6 while halving QPS (63.3 → 37.6); larger vocabularies help overall but slightly underperform moderate sizes on visually fine-grained tasks (CIRR, NIGHTS). Larger decoders improve text-centric tasks (WebQA R@1: 20.7 → 24.1 from T5-small to T5-large) but hurt compositional visual tasks (CIRR R@1: 14.4 → 10.4), indicating the bottleneck there lies in identifier construction rather than decoder capacity. The global prior weight $\lambda\phi$5 is stable across tasks, though optimal values are task-dependent.

Qualitative case studies illustrate three regimes: dual-guided generation succeeding directly, beam-search errors corrected by dense reranking, and a residual failure mode where neither the discrete identifier nor the dense reranker distinguishes candidates requiring joint visual–textual evidence reasoning (e.g., identifying a specific factual detail about a visually similar scene).

Limitations and open questions

The paper is explicit about several constraints. The effectiveness gap to strong dense and LMM-based retrievers persists, particularly on knowledge-intensive text matching, because short discrete identifiers discard fine-grained semantic information. The framework uses a staged training pipeline—representation learning, quantization, set-based scoring, and decoder training are optimized separately—and whether end-to-end coupling would improve identifier-retrieval alignment remains untested. Dynamic corpora are not addressed: how to handle candidate insertion, deletion, and identifier updates without retraining is left open. Finally, evaluation is confined to image–text benchmarks; web-scale corpora, video retrieval, and retrieval-augmented multimodal generation remain unexamined.

Conclusion

DrIG demonstrates that a single residual-quantized identifier can serve simultaneously as an autoregressive generation target and a prefix-independent relevance signal, substantially narrowing—but not closing—the gap between generative and dense universal multimodal retrieval. The consistent +27–29% relative improvement over GENIUS, near-zero-shot competitiveness on Flickr30K, and stable throughput under corpus growth establish dual-role identifiers as a practical design for scalable generative multimodal retrieval, provided hybrid dense reranking is available to recover quantization losses.