---
title: 'Omni-Embed-Audio: Robust Audio-Text Retrieval'
url: https://www.emergentmind.com/topics/omni-embed-audio-oea
type: topic
---

# Omni-Embed-Audio: Robust Audio-Text Retrieval

Omni-Embed-Audio (OEA) is a retrieval-oriented multimodal encoder that leverages multimodal LLMs with native audio understanding to map text queries and audio clips into a shared 512-dimensional embedding space for text-to-audio and text-to-text retrieval [2604.18360]. Its central claim is not that audio-text retrieval is unsolved on caption benchmarks, but that existing evaluation protocols overestimate robustness because they rely on caption-style queries that closely match training data, whereas practical search involves questions, commands, keyword tags, paraphrases, and exclusion-based requests [2604.18360]. OEA is instantiated with Omni-Embed-Nemotron-3B, Qwen2.5-Omni-3B, and Qwen2.5-Omni-7B backbones, and can also be understood within a broader shift toward unified audio embeddings inside omni-modal retrieval systems, including shared-space audio branches in Omni-Embed-Nemotron [2510.03458].

## 1. Problem formulation and motivation

OEA is proposed against two limitations of current audio-text retrieval practice. First, most modern audio-language models are based on Contrastive Language-Audio Pretraining (CLAP), are evaluated on benchmarks such as AudioCaps and Clotho, and are typically trained and tested on caption-style queries such as “A dog is barking in a large echoing hall.” Second, those benchmarks mirror the training distribution, so apparent performance can overestimate robustness under realistic search behavior [2604.18360].

The benchmark mismatch is explicit. Real search queries in Freesound logs are much shorter—on average 1.8 tokens—and are often keyword-like rather than caption-like. The same analysis also cites prior work showing performance drops of up to 16–20% when queries are paraphrased or reformulated. OEA therefore treats robustness to query variation and exclusion as first-order evaluation targets rather than secondary stress tests [2604.18360].

Within this formulation, a robust audio retrieval system must satisfy two requirements. It must preserve semantic intent under varied language forms such as questions or imperatives, and it must respect exclusions such as “without thunder” or “not engine noise.” Standard caption-based retrieval metrics test neither requirement directly. OEA’s contribution is to recast audio-text retrieval around these failures and to supply both a retrieval architecture and an evaluation framework aligned with that reframing [2604.18360].

## 2. Model architecture and optimization

OEA uses a single shared transformer backbone for both modalities rather than separate text and audio encoders. Text queries are wrapped in a chat-style template with a `query:` prefix, tokenized, passed through the transformer, and the last hidden layer is mean-pooled. Audio uses the backbone’s native audio encoder on raw waveform input, formatted with a `passage:` prefix, and the last hidden layer is likewise mean-pooled. Each modality then passes through its own projection head: a bias-free linear layer from hidden dimension to 512, followed by dropout, LayerNorm, and L2 normalization, yielding unit-norm embeddings \(t_i, a_i \in \mathbb{R}^{512}\) [2604.18360].

Because the embeddings are L2-normalized, OEA uses cosine similarity,
\[
\text{sim}(t_i, a_j) = t_i^\top a_j .
\]
Training uses symmetric InfoNCE contrastive loss,
\[
\mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{t \rightarrow a} + \mathcal{L}_{a \rightarrow t}\right),
\]
with
\[
\mathcal{L}_{t \rightarrow a} = -\log \frac{\exp(\text{sim}(t_i, a_i)/\tau)}{\sum_j \exp(\text{sim}(t_i, a_j)/\tau)},
\]
and analogously for \(\mathcal{L}_{a \rightarrow t}\), using \(\tau = 0.07\) [2604.18360].

The backbones are frozen. OEA trains only LoRA adapters on attention projections—query, key, value, and output—with rank \(r = 16\), scaling \(\alpha = 32\), and dropout \(0.05\), together with the modality-specific projection heads. The trainable parameter budget is around 11–16M parameters, corresponding to 0.29–0.36% of total backbone parameters. This parameter-efficient scheme is intended to preserve the backbone’s general audio understanding and instruction-following behavior while adapting it for retrieval [2604.18360].

The training curriculum is staged. Initial audio-text alignment uses WavCaps with 275,618 samples of up to 31 seconds. Caption-based retrieval is then refined on AudioCaps v2 with 91,256 training samples. An optional extra stage on Clotho v2 with 3,839 clips improves performance on natural captions; models with this additional stage are denoted “OEA (+Cl).” No explicit multi-task loss is added for text-to-text retrieval: it emerges because both query text and caption text pass through the same backbone and projection head [2604.18360].

## 3. User-Intent Queries and hard-negative evaluation

OEA’s evaluation framework introduces User-Intent Queries (UIQs), a five-way typology designed to mimic actual search behavior. The five types are Question Queries, Imperative Queries, Keyphrase Queries, Paraphrase Queries, and Negative Queries. Question Queries are 8–18 words, begin with forms such as “Can you” or “Do you,” and end with a question mark. Imperative Queries are 8–15 words, begin with “Find,” “Search for,” “Locate,” or “Retrieve,” and contain no question mark. Keyphrase Queries are 3–6 comma-separated lowercase tags. Paraphrase Queries are declarative reformulations without command or question tone. Negative Queries explicitly describe desired content while excluding features from a hard negative, such as “Heavy rain and wind on metal surfaces without thunder or engine noise” [2604.18360].

The resulting benchmark comprises AudioCaps-UIQ, Clotho-UIQ, and MECAT-UIQ, totaling 13,053 queries: 4,530 for AudioCaps, 4,722 for Clotho, and 3,801 for MECAT. Positive UIQs are generated with GPT-5.1 under constraints that reuse wording from original captions and keep length within \(\pm 2\) words of the original caption. Human validation uses 9 annotators, 75 samples, and 675 ratings on a 1–5 Likert scale, yielding an overall mean of 4.15/5.0; Claude Opus 4.5 gives 4.43/5.0. By query type, human ratings are 4.26 for Question, 4.16 for Imperative, 4.35 for Keyphrase, 4.14 for Paraphrase, and 3.82 for Negative [2604.18360].

Negative-query evaluation depends on a dedicated hard negative mining pipeline. For each target audio, Stage 1 retrieves top-\(K\) acoustically similar candidates with \(K=20\) using MGA-CLAP audio embeddings. Stage 2 applies an acoustic similarity threshold. Stage 3 computes semantic similarity between target and candidate captions using BGE-large-en-v1.5 sentence embeddings. Stage 4 keeps acoustically similar but semantically distinct pairs, and a final human verification pass removes non-confusable artifacts. The objective is to create target–hard-negative pairs that are genuinely difficult from an auditory standpoint while supporting explicit exclusion language [2604.18360].

To score these cases, OEA supplements Recall@\(k\) with discrimination metrics. Hard Negative Suppression Rate at top \(k\) is defined as
\[
\text{HNSR@k} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\{\text{rank}(T_i) \le k \text{ and } \text{rank}(HN_i) > k\},
\]
and Target-First Rate at top \(k\) is
\[
\text{TFR@k} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\{\text{rank}(T_i) = 1 \text{ and } \text{rank}(HN_i) > k\}.
\]
The benchmark also reports global HNSR and \(\Delta\)-Rank, where
\[
\Delta\text{-Rank}_i = \text{rank}(HN_i) - \text{rank}(T_i).
\]
These metrics measure whether the system both retrieves the target and suppresses the acoustically similar distractor [2604.18360].

## 4. Empirical results

OEA’s results separate four regimes: caption-style text-to-audio retrieval, caption-based text-to-text retrieval, UIQ robustness, and exclusionary-query discrimination. Across these regimes, its main empirical pattern is consistent: it is competitive rather than dominant on in-domain caption retrieval, but stronger when semantic query understanding and distractor suppression matter most [2604.18360].

| Setting | Representative result | Interpretation |
|---|---|---|
| Caption-style T2A | AudioCaps: M2D-CLAP \(R@5 = 77.13\%\); best OEA (Nemo3B) \(R@5 = 72.64\%\) | Slightly lower on in-domain caption retrieval |
| Cross-domain / leakage-free T2A | Clotho: OEA-Qwen3B (+Cl) \(R@10 = 63.25\%\) vs M2D-CLAP \(55.54\%\); MECAT: OEA-Nemo3B \(R@5/R@10 = 24.53\% / 35.81\%\) | Competitive or superior on cross-domain settings |
| Caption-based T2T | Clotho: OEA-Qwen3B (+Cl) \(R@1/R@5/R@10 = 64.52\% / 75.25\% / 79.71\%\) vs M2D-CLAP \(55.85\% / 69.05\% / 74.76\%\) | Dominant text-to-text retrieval |
| UIQ robustness | Mean UIQ: M2D-CLAP \(47.76\%\), OEA-Qwen3B (+Cl) \(47.18\%\); Imperative: OEA-Qwen7B (+Cl) \(49.87\%\) vs M2D-CLAP \(44.74\%\) | Strongest gains on command-style queries |
| Negative-query discrimination | HNSR@10: OEA-Qwen7B (+Cl) \(34.6\%\) vs M2D-CLAP \(30.3\%\); HNSR: \(74.4\%\) vs \(68.0\%\); TFR@10: OEA-Nemo3B (+Cl) \(10.1\%\) vs \(7.5\%\) | Better suppression of hard negatives |

On AudioCaps, the strongest CLAP baseline remains ahead in caption-style retrieval: M2D-CLAP reaches \(R@5 = 77.13\%\), whereas the best OEA result, Nemo3B, reaches \(72.64\%\). On the mean across all datasets, M2D-CLAP reaches mean T2A \(R@5 = 47.86\%\), while OEA-Qwen7B (+Cl) reaches \(46.40\%\). On Clotho and MECAT, however, OEA closes or reverses the gap. OEA-Qwen3B (+Cl) reaches \(R@10 = 63.25\%\) on Clotho, which is +7.71 points over M2D-CLAP’s \(55.54\%\), and OEA-Nemo3B achieves the best MECAT \(R@5/R@10 = 24.53\% / 35.81\%\) [2604.18360].

The most pronounced gains appear in caption-based text-to-text retrieval. On Clotho, OEA-Qwen3B (+Cl) attains \(R@1 = 64.52\%\), \(R@5 = 75.25\%\), and \(R@10 = 79.71\%\), compared with M2D-CLAP’s \(55.85\%\), \(69.05\%\), and \(74.76\%\). On MECAT, OEA-Qwen7B (+Cl) raises \(R@5\) from 38.74% to 47.41%, a relative improvement of approximately 22%. The paper’s interpretation is that the unified LLM backbone yields dominant text-to-text retrieval performance, especially in caption-indexed pipelines [2604.18360].

Under UIQs, overall average performance remains close to the best CLAP model, but the distribution of strengths shifts. OEA-Qwen3B (+Cl) achieves an overall Avg UIQ score of 47.18%, compared with 47.76% for M2D-CLAP. The most visible OEA advantage appears on imperative queries: OEA-Qwen7B (+Cl) reaches 49.87%, versus 44.74% for M2D-CLAP, a +5.13 point gain. This result is consistent with the backbone’s instruction-tuned language modeling heritage [2604.18360].

Negative-query evaluation sharpens the contrast. Standard negative-query retrieval remains effectively tied, with M2D-CLAP at \(R@5 = 41.56\%\) and OEA-Qwen3B (+Cl) at \(41.52\%\). Yet OEA performs better on the discrimination metrics that measure suppression rather than mere retrieval: HNSR@10 improves from 30.3% to 34.6%, global HNSR from 68.0% to 74.4%, and TFR@10 from 7.5% to 10.1%, a relative improvement of approximately 34.7% [2604.18360].

## 5. Relation to omni-modal retrieval and embedding research

OEA is retrieval-specialized, but it belongs to a larger family of omni-modal models that already encode audio into shared spaces. Omni-Embed-Nemotron, for example, is a bi-encoder retrieval model built on the Qwen2.5-Omni Thinker component, where raw audio is converted into audio tokens, processed by the same multimodal encoder used for text, image, and video, pooled into a single embedding vector, and compared with dot product or cosine similarity in a shared embedding space. In that model, audio and video are encoded separately for retrieval, and audio+video documents are most effective under late fusion rather than token-level interleaving for FineVideo and LPM [2510.03458]. OEA can therefore be read as a dedicated audio-text retrieval formulation built on the same broader transition from modality-specific towers toward shared-space omni-modal encoders.

A related line of work addresses alignment issues that OEA itself does not explicitly target. e5-omni isolates modality-dependent similarity sharpness, mixed-modality negative imbalance, and mismatched embedding statistics as central problems in omni-modal retrieval. Its recipe combines modality-aware temperature calibration, a controllable negative curriculum with debiasing, and batch whitening with covariance regularization, and raises AudioCaps Recall@1 to 37.7 for e5-omni-7B [2601.03666]. This line is complementary to OEA: where OEA redesigns evaluation around real search behavior and exclusion, e5-omni redesigns optimization and geometry around explicit cross-modal alignment.

Broader benchmark work also clarifies what OEA does not solve. MMEB-V3 evaluates full-modality embeddings across text, image, video, audio, and agent-centric tasks, and finds that models often fail to retrieve the intended target modality, that cross-modal retrieval is highly asymmetric and dominated by query-modality bias, and that instruction-induced shifts are either insufficient or misaligned with the target modality [2604.23321]. Those findings imply that robust audio-text retrieval under UIQs does not automatically imply modality-aware omni-modal retrieval under explicit target-modality constraints. OEA improves semantic handling of audio-related queries; it does not claim to solve full-modality instruction compliance.

## 6. Limitations and open research questions

OEA inherits several limitations directly from its design. It currently depends on multimodal LLMs with native audio encoders; extending the approach to text-only LLMs with separate audio front-ends is identified as future work. It also uses larger 3B–7B backbones, with higher memory use than CLAP, so practical deployment may require quantization or distillation for edge devices. Its hard negative set is mined with MGA-CLAP, BGE-large-en-v1.5, and human verification, but this procedure may still miss other forms of acoustic confusion. Its UIQs are generated by GPT-5.1 under constraints and, although validated, may not cover the full variety of real-world queries [2604.18360].

A broader limitation emerges when OEA is viewed as a candidate component in a universal audio embedding stack. MAEB evaluates 50+ models across 30 tasks spanning speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages, and reports that no single model dominates across all tasks. Contrastive audio-text models excel on environmental sound classification but score near random on multilingual speech tasks, speech-pretrained models show the opposite pattern, and clustering remains difficult for all models, with even the best-performing model achieving only modest results [2602.16008]. This broader evidence indicates that OEA’s gains in semantic query robustness and hard-negative suppression should not be conflated with universal audio embedding competence.

The main research direction suggested by this constellation of results is therefore twofold. One direction is to extend OEA’s robustness program—natural query forms, negation, exclusion, distractor suppression—into wider omni-modal settings. The other is to combine that program with stronger cross-modal geometry, multilingual speech coverage, and broader non-speech audio coverage so that improvements on semantic query interpretation do not come at the cost of speech, music, environmental sound, or clustering performance. In that sense, OEA is best understood as a retrieval-oriented intervention into the query semantics of audio-text search, not as the final form of an omni-purpose audio embedding model [2604.18360].

Source: https://www.emergentmind.com/topics/omni-embed-audio-oea