OmniRetrieval: Unified Heterogeneous Data Retrieval

• OmniRetrieval is a unified retrieval approach that preserves the distinct characteristics of heterogeneous data sources through native query execution.
• It integrates dense, sparse, and multimodal embedding techniques to support diverse applications, from memory-centric systems to industrial pipelines.
• By deferring source commitment and orchestrating multiple retrieval channels, OmniRetrieval enhances accuracy and adaptability across benchmarks.

OmniRetrieval denotes a class of retrieval systems that aim to provide a unified interface to heterogeneous information while preserving the distinctions that make different retrieval substrates useful. In the narrow sense, it is the name of a framework for unified retrieval across unstructured corpora, relational databases, RDF knowledge graphs, and labeled property graphs, where a natural-language query is routed to source-native execution engines rather than forced into a single homogenized index (Baek et al., 28 May 2026). In a broader research sense, the term also captures related efforts to build one retrieval backbone across knowledge-intensive NLP tasks (Maillard et al., 2021), one embedding space across multiple modalities (Liu et al., 26 May 2026, Xu et al., 3 Oct 2025, Huynh et al., 2 Mar 2026), one memory substrate for long-horizon agents (Liu et al., 1 Apr 2026), or one operational retrieval layer for industrial systems (Zhang et al., 11 Jun 2026).

1. Conceptual scope

Recent work uses “OmniRetrieval” in several related but non-identical senses. One line seeks a single retriever across many tasks over a shared corpus, as in multi-task dense retrieval for KILT-style knowledge-intensive NLP (Maillard et al., 2021). Another seeks one model with dual dense and lexicon-based outputs, so that semantic and lexical retrieval become two views of the same encoder (Shen et al., 2022). A third treats unification as orchestration across heterogeneous backends, with source selection, source-native query generation, and native execution rather than a shared embedding space (Baek et al., 28 May 2026). A fourth seeks a unified multimodal embedding space supporting text, image, video, and audio retrieval, sometimes including any-to-any or composed-query retrieval (Liu et al., 26 May 2026, Xu et al., 3 Oct 2025, Huynh et al., 2 Mar 2026).

A common misconception is that omni-retrieval necessarily means collapsing everything into one representation. The heterogeneous-source formulation argues the opposite: collapsing relational tables, RDF graphs, property graphs, and text corpora into a shared space erases schemas, ontologies, joins, and traversal operators, so effective unification may require an orchestration layer above the sources rather than homogenization within them (Baek et al., 28 May 2026). A second misconception is that unification always implies one fixed retriever. Query-adaptive mixtures show that heterogeneous retrievers can remain complementary at the query level, and that a unified retrieval layer may instead estimate which retrievers to trust per query (Kalra et al., 18 Jun 2025).

2. Antecedents in universal text retrieval

An important precursor is “universal” dense retrieval for knowledge-intensive NLP. “Multi-task Retrieval for Knowledge-Intensive Tasks” builds directly on DPR with a shared query encoder, a shared passage encoder, a 36 million-passage KILT Wikipedia index, inner-product similarity, BM25 hard negatives, and FAISS search, then trains one retriever on the union of FEVER, AIDA-YAGO2, T-REx, zsRE, Natural Questions, HotpotQA, TriviaQA, and Wizard of Wikipedia (Maillard et al., 2021). Its main significance is breadth rather than narrow per-task optimality: a single retriever remains competitive with task-specific systems across fact checking, entity linking, slot filling, question answering, and dialogue grounding, and the few-shot results argue that retrieval competence itself can be pretrained across tasks rather than relearned from BERT initialization for each target task (Maillard et al., 2021).

A second antecedent is unification across retrieval paradigms rather than tasks. “UnifieR: A Unified Retriever for Large-Scale Retrieval” defines one encoder with dual-representing capability, outputting both a dense sequence vector and a sparse lexicon-weighting vector, then trains the dense and lexical branches jointly with contrastive loss, cross-paradigm hard negatives, agreement regularization, and FLOPS sparsity regularization (Shen et al., 2022). Its uni-retrieval scheme uses lexical retrieval to generate candidates and dense scoring to rerank them, showing that “unified retrieval” can mean dense–sparse co-design inside one model rather than simple score fusion between separately trained systems (Shen et al., 2022).

A third line questions the premise that one retriever should dominate all queries. “MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers” treats retrieval as a per-query allocation problem over heterogeneous retrievers and retrieval granularities. Scores are normalized to $[0,1]$, query-specific trust weights are estimated from unsupervised geometric signals, and a weighted mixture is formed over sparse, dense, multi-granular, and even human retrievers (Kalra et al., 18 Jun 2025). This suggests that omni-retrieval can also be understood as a unified control layer over multiple retrieval channels rather than a monolithic encoder.

3. Source-native retrieval across heterogeneous knowledge bases

In its strictest formulation, OmniRetrieval is a framework for natural-language access to structurally heterogeneous knowledge bases. The system maintains a pool of sources $\mathcal{B}=\{b_1,\ldots,b_N\}$, each with structural context $c_b$ and a native execution engine. Given a question $q$, it first selects a subset of candidate sources,

$$\mathcal{S} = LLM_{select}(q,\{c_b\}_{b\in\mathcal{B}},k)\subseteq\mathcal{B},$$

then generates a source-native query for each selected source,

$\hat{q}_b = Generate_b(q,c_b),$

executes these queries in native engines, and finally performs cross-source evidence selection,

$$\mathcal{E} = Select(q,\{Exec(b,\hat{q}_b)\}_{b\in\mathcal{S}}).$$

The supported backends are unstructured corpora, relational databases queried with SQL, RDF knowledge graphs queried with SPARQL, and labeled property graphs queried with Cypher (Baek et al., 28 May 2026).

The benchmark spans 13 datasets and 309 distinct knowledge bases: 7 document-search corpora, 286 relational databases from Spider and BIRD, 1 RDF knowledge base through Wikidata, and 15 property graphs from the Neo4j collection (Baek et al., 28 May 2026). Across backbone models, OmniRetrieval improves over the stronger one-shot routing baseline, KB Routing, on Source Selection Accuracy (65.71 vs. 61.65), Retrieval Accuracy (44.34 vs. 39.98), and LLM-as-a-Judge (65.88 vs. 57.99) (Baek et al., 28 May 2026). The paper’s central claim is therefore not merely that multiple sources help, but that deferred commitment matters: the framework selects several plausible sources, issues source-native queries to each, and decides after execution rather than before it (Baek et al., 28 May 2026).

This formulation sharpens a major theoretical distinction within the field. Unified-representation retrieval is attractive because it promises one index, but the constrained comparison in the paper shows that a unified representation baseline remains well below OmniRetrieval. The stated reason is structural: atomic-item retrieval in a common space cannot express joins, ontological matching, or path traversal in the same way that SQL, SPARQL, and Cypher can (Baek et al., 28 May 2026). In this sense, source-native orchestration is not an implementation detail; it is the paper’s definition of what a heterogeneous omni-retrieval layer should preserve.

4. Multimodal and any-to-any formulations

A second major branch of OmniRetrieval research seeks one embedding space across modalities. “OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation” trains a unified audio-video-text encoder with three losses: pairwise alignment $\mathcal{L}_A$, fusion-as-teacher distillation $\mathcal{L}_D$, and Tuple-InfoNCE $\mathcal{L}_T$, yielding the full objective $\mathcal{B}=\{b_1,\ldots,b_N\}$0 with equal coefficients (Liu et al., 26 May 2026). It evaluates 12 retrieval directions, including single-to-single and single-to-dual settings such as $\mathcal{B}=\{b_1,\ldots,b_N\}$1, $\mathcal{B}=\{b_1,\ldots,b_N\}$2, and $\mathcal{B}=\{b_1,\ldots,b_N\}$3, and reports AVG-all 34.84 on OmniRetriever-Bench, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03 (Liu et al., 26 May 2026).

“Omni-Embed-Nemotron” adapts the Qwen2.5-Omni Thinker backbone into a bi-encoder retrieval model over text, image, audio, and video, trained with InfoNCE and hard-negative mining while freezing the audio and visual encoders and applying LoRA only to the LLM (Xu et al., 3 Oct 2025). Its most notable architectural result is that retrieval benefits from separate-stream treatment of audio and video rather than the interleaved fusion used in the original generative model; on both LPM and FineVideo, “Audio+Video Separately” outperforms “Audio+Video Fusion” (Xu et al., 3 Oct 2025). “OmniRet” pushes further toward composed tri-modal retrieval by combining text, vision, and audio in one model, adding a Shared Media Resampler for token compression and Attention Sliced Wasserstein Pooling for higher-fidelity single-vector aggregation, and training on approximately 6 million query-target pairs spanning 30 datasets (Huynh et al., 2 Mar 2026).

A third multimodal route makes reasoning itself part of retrieval. “Retrv-R1” treats universal multimodal retrieval as a two-stage process: an embedding retriever produces top-$\mathcal{B}=\{b_1,\ldots,b_N\}$4 candidates, and a reasoning-oriented MLLM reranks them using an Information Compression Module, a Details Inspection Mechanism, synthetic retrieval CoT, and GRPO with a curriculum reward that penalizes unnecessary inspections (Zhu et al., 3 Oct 2025). The result is a universal query-to-item selector supporting text, image, and interleaved multimodal queries, with especially strong gains on hard multi-input retrieval settings (Zhu et al., 3 Oct 2025). In a more domain-specialized variant, “ReID5o” formulates Omni Multi-modal Person Re-identification as retrieval from an RGB gallery using arbitrary combinations of infrared, color pencil, sketch, and text queries, and uses a unified CLIP-based encoder with multi-expert routing and feature mixture to support single-, dual-, tri-, and quad-modal search in one model (Zuo et al., 11 Jun 2025).

Query robustness is itself an omni-retrieval problem. “Omni-Embed-Audio” shows that caption-style audio benchmarks mask failures on realistic user queries, introducing User-Intent Queries with questions, commands, keyword tags, paraphrases, and exclusion-based negatives, together with hard-negative discrimination metrics such as HNSR and TFR (Yoo et al., 20 Apr 2026). This suggests that omni-retrieval is not only about modality coverage, but also about robustness to heterogeneous query formulations.

5. Memory-centric and temporal OmniRetrieval

In memory systems, omni-retrieval shifts from modality unification to retrieval over multiple memory substrates and timescales. “Omni-SimpleMem” defines a Multimodal Atomic Unit,

$\mathcal{B}=\{b_1,\ldots,b_N\}$5

where $\mathcal{B}=\{b_1,\ldots,b_N\}$6 is a summary, $\mathcal{B}=\{b_1,\ldots,b_N\}$7 an embedding, $\mathcal{B}=\{b_1,\ldots,b_N\}$8 a cold-storage pointer, $\mathcal{B}=\{b_1,\ldots,b_N\}$9 a timestamp, $c_b$0 a modality tag, and $c_b$1 structural links (Liu et al., 1 Apr 2026). Retrieval is hybrid dense+sparse with rank-preserving set-union fusion,

$c_b$2

then pyramid retrieval expands evidence from summaries to full text/captions to raw assets under a token budget, and a knowledge graph adds relational recall (Liu et al., 1 Apr 2026). The paper’s practical lesson is that omni-retrieval over memory is layered: dense similarity, lexical backoff, graph expansion, and coarse-to-fine evidence loading are all part of the same retrieval pipeline.

“OmniQuery” performs a related transformation for personal media archives by augmenting photos, screenshots, and videos with atomic context, composite context, and semantic knowledge before retrieval (Li et al., 2024). Queries are rewritten, decomposed, and expanded, then matched against multiple stores: structured memories, composite contexts, and inferred semantic knowledge (Li et al., 2024). “Omni-RAG” makes the same point in live open-domain RAG: retrieval quality is often dominated by pre-retrieval query handling, so the framework rewrites noisy inputs, decomposes multi-intent queries, retrieves per sub-query from FineWeb via OpenSearch, reranks with BGE, and only then generates a final answer (Dong et al., 26 Jun 2025).

The memory perspective also extends to internal model state. “O-Mem” organizes long-horizon agent memory into persona memory, working memory, and episodic memory, then retrieves from all three in parallel: $c_b$3 Its active user profiling mechanism incrementally extracts topics, attributes, and events from each interaction, then supports hierarchical retrieval of persona attributes and topic-related context (Wang et al., 17 Nov 2025). “OmniMem,” in turn, treats the historical KV cache in autoregressive video generation as an explicit retrieval store rather than an implicit state, using Adaptive Window Exclusion, Query-Shared KV Selection, and Per-Head Scattered KV Access to retrieve sparse full-resolution KV blocks from long-range history (Zhao et al., 28 May 2026). Both systems imply that omni-retrieval can target not only external corpora but also agent memory and model-internal history.

6. Industrial and open-world operationalization

Industrial retrieval systems sharpen the operational requirements of omni-retrieval. “OneRetrieval” addresses a large e-commerce stack in which sparse inverted-index retrieval, dense retrieval, and collaborative retrieval were previously separate branches. Its solution is a single generative retrieval model built on Keyword-Aligned Encoding, where each identifier position is tied to interpretable attribute words, six merged codebook groups, and reserved slots that can be bound to new terms after deployment without retraining (Zhang et al., 11 Jun 2026). The key industrial constraint is editability: the system is designed so that operators can inject a new term, map it deterministically to a reserved slot, and route it to target items without modifying model parameters (Zhang et al., 11 Jun 2026). In other words, omni-retrieval in production must preserve not only recall quality but also operational control.

Open-world tool use introduces another operational axis: retrieval must feed action. “ToolOmni” treats tool use as a retrieval-and-execution problem over a repository of 16,464 APIs across 49 categories, where the agent must proactively search, refine its search queries, select a task-complete subset of tools, and then execute them under feedback (Huang et al., 15 Apr 2026). The method combines cold-start SFT with a Decoupled Multi-Objective GRPO algorithm that jointly optimizes tool retrieval accuracy and grounded execution, and it reports a +10.8% gain in end-to-end execution success rate over strong baselines (Huang et al., 15 Apr 2026). This suggests that in agent settings, retrieval ceases to be a front-end service and becomes a sub-policy inside the reasoning loop.

These industrial and agentic systems also reveal a persistent design split. One branch seeks a single editable or executable retrieval model, as in OneRetrieval and ToolOmni. Another keeps multiple retrievers but unifies them through adaptive control, as in mixture-of-retrievers frameworks (Kalra et al., 18 Jun 2025). The resulting tension—monolithic unification versus orchestrated heterogeneity—has become one of the field’s central design questions.

7. Persistent limits and open research questions

Taken together, these works suggest that omni-retrieval is feasible but still bounded. A recurring limitation is domain restriction: early “universal” dense retrieval remains tied to one shared Wikipedia corpus and supervised KILT alignments rather than arbitrary source mixtures (Maillard et al., 2021). Multimodal unified encoders remain uneven: Omni-Embed-Nemotron is competitive rather than dominant on pure text retrieval and trails specialized visual-document retrievers on ViDoRe (Xu et al., 3 Oct 2025), while OmniRet is strongest on composed, video, and audio tasks but weaker on image-to-image retrieval than specialist visual models (Huynh et al., 2 Mar 2026).

A second open question is where unification should happen. Heterogeneous-source orchestration argues against collapsing all sources into one representation because doing so destroys schemas, ontologies, joins, and graph traversal (Baek et al., 28 May 2026). Multimodal embedding models argue the opposite for audio, video, text, and images, where shared spaces are precisely what enable any-to-any retrieval (Liu et al., 26 May 2026). This suggests that “omni” may remain a layered property rather than a single recipe: shared embeddings for some modalities, source-native execution for some backends, and adaptive mixtures for some query regimes.

A third issue is the trade-off between fidelity, abstraction, and efficiency. Memory systems repeatedly show that compressed summaries, persona abstractions, or pooled embeddings can improve latency and control token budgets, yet they can also lose nuance, rely on metadata correctness, or introduce retrieval blind spots (Liu et al., 1 Apr 2026, Wang et al., 17 Nov 2025). The same tension appears in multimodal retrieval, where token compression and single-vector pooling are necessary for scalability but must be compensated by mechanisms such as ASWP, retrieval-specific reasoning, or selective inspection (Huynh et al., 2 Mar 2026, Zhu et al., 3 Oct 2025).

A final question is evaluation. Standard recall metrics remain useful, but recent work increasingly exposes failures that classic benchmarks miss: query-style robustness, exclusion handling, hard-negative suppression, cross-source evidence selection, editability, and execution success (Yoo et al., 20 Apr 2026, Baek et al., 28 May 2026, Zhang et al., 11 Jun 2026, Huang et al., 15 Apr 2026). This suggests that OmniRetrieval is not only a modeling problem but also a benchmark design problem. A plausible implication is that future systems will be judged less by one leaderboard on one corpus and more by how well they preserve structure, cover modalities, adapt to evolving repositories, and support downstream reasoning or action across heterogeneous environments.