AuroLA: Audio–Text Retrieval Framework

• AuroLA is a unified audio–text retrieval framework that leverages an MLLM backbone to generate prompted, contextualized embeddings for enhanced semantic matching.
• It integrates a scalable data pipeline with multi-granular supervision via long descriptions, tags, and event segments to refine audio–text alignment.
• AuroLA employs a two-stage process combining embedding-based retrieval with bidirectional MLLM-based cross-modal re-ranking, achieving superior performance with minimal training data.

AuroLA is a contrastive language–audio pre-training framework for audio–text retrieval that repurposes a Multimodal LLM (MLLM) as a unified backbone for both embedding-based retrieval and cross-modal re-ranking. It is designed to address a central limitation of classic dual-encoder systems such as CLAP: small-scale text encoders often struggle with complex queries that require reasoning or world knowledge. In AuroLA, audio and text are summarized under instruction-style prompts, encoded through special retrieval tokens, optimized with a multi-granular contrastive objective, and optionally refined by bidirectional MLLM-based re-ranking. The framework is reported to consistently outperform state-of-the-art models, including PE-AV, while utilizing only approximately $1\%$ of PE-AV’s training data, and it exhibits clear scaling trends with respect to dataset size and model capacity (Xu et al., 20 Feb 2026).

1. Conceptual position in audio–text retrieval

AuroLA is situated within the audio–text retrieval problem, where audios and texts are mapped into a shared semantic space so that semantically aligned pairs lie close under a similarity function. The immediate predecessor paradigm is the contrastive dual encoder, in which an audio encoder and a text encoder are trained with paired audio–caption data. The AuroLA formulation retains the contrastive retrieval setting but replaces the conventional pair of small encoders with an MLLM used as a unified backbone (Xu et al., 20 Feb 2026).

The motivating diagnosis is that standard dual encoders can be limited in compositionality, reasoning, and world-knowledge grounding. Queries such as “a slow build-up of thunder followed by rain on a tin roof at night” or “a chef slicing vegetables and sizzling pan ambience in a crowded kitchen” require more than keyword matching; they require integration of temporal structure, source attribution, and contextual knowledge. AuroLA addresses this by exploiting the knowledge-rich language priors and cross-modal reasoning capacity associated with MLLMs. This suggests a shift from shallow alignment of captions and clips toward retrieval models that can summarize, compress, and compare audio and language in a more semantically structured manner (Xu et al., 20 Feb 2026).

AuroLA uses a two-stage retrieval design. The first stage performs embedding retrieval by encoding audio and text into a shared space and ranking by cosine similarity. The second stage applies bidirectional re-ranking, using the same MLLM in a cross-attention mode to rescore top-$k$ candidates through deeper audio–text interaction (Xu et al., 20 Feb 2026).

2. Data pipeline and multi-granular supervision

AuroLA includes a scalable data pipeline that curates diverse audio from multiple sources and generates multi-granular captions via automated annotation (Xu et al., 20 Feb 2026). The audio sources described for this pipeline include both curated libraries and academic datasets. Library-style sources include BBC Sound Effects, Freesound, FreeToUseSounds, Sonniss GameAudioGDC, Audiostock, SoundBible, and Epidemic Sound. Academic sources include AudioCaps, Clotho, WavCaps, AudioSetCaps, FusionAudio-1.2M, AudioSet, ESC-50, and VGGSound, with related audiovisual corpora such as Epic-Sounds and HD-EPIC used for event inventories and longer-duration contexts (Xu et al., 20 Feb 2026).

The supervision is explicitly multi-granular. Long descriptions provide narrative captions that can include sources, actions, environments, temporal evolution, emotions, and onomatopoeia. Structured tags and keywords are drawn from ontology labels, library metadata, and audio tagging models. Event-level annotations provide segment-level labels or pseudo-labels for temporal events. This organization is central to the later training objective, because different granularities are aligned jointly rather than collapsed into a single caption format (Xu et al., 20 Feb 2026).

The pipeline also includes cleaning, deduplication, and quality control. The described operations include metadata normalization, acoustic deduplication through audio fingerprinting, duration and signal-to-noise filtering, language filtering, and mapping of library tags to AudioSet ontology identifiers. These steps are important because noisy tags, duplicated clips, and ambiguous metadata are all identified as failure modes for naive contrastive training. A plausible implication is that AuroLA’s performance depends not only on MLLM capacity but also on the structure and reliability of the supervision signal (Xu et al., 20 Feb 2026).

3. MLLM adaptation, prompting, and embedding construction

AuroLA adapts an MLLM for retrieval by prompting it to summarize the audio or text input and by using the hidden state of a special token as the retrieval embedding (Xu et al., 20 Feb 2026). The description does not specify an exact backbone, but it frames the architecture as conceptually similar to systems with an audio token interface, including Qwen-Audio, Qwen-Omni, Audio Flamingo, and MLLM embedding variants such as E5-v and Qwen3-VL-Embedding (Xu et al., 20 Feb 2026).

On the audio side, raw waveforms are converted to mel-spectrogram patches, and a small CNN or ViT-style patchifier produces a sequence of audio tokens that are projected into the MLLM hidden dimension. On the text side, the backbone tokenizer is used in the ordinary way. The key mechanism is the insertion of special tokens, denoted $[\mathrm{AUD\text{-}EMB}]$ and $[\mathrm{TXT\text{-}EMB}]$, whose final-layer hidden states are passed through small projection heads to define the retrieval embeddings. The embedding equations are given as

$$e_a = \mathrm{norm}\!\big(W_a\, h([\mathrm{AUD\text{-}EMB}])\big), \qquad e_t = \mathrm{norm}\!\big(W_t\, h([\mathrm{TXT\text{-}EMB}])\big),$$

where $h(\cdot)$ is the final-layer hidden state, $W_a$ and $W_t$ are learned linear projections, and $\mathrm{norm}(\cdot)$ is $L_2$-normalization (Xu et al., 20 Feb 2026).

Prompting is part of the architecture rather than an auxiliary convenience. The audio summarization prompt is described as instruction-style prompting that asks the model to summarize source(s), event(s), scene context, and notable acoustic attributes. The text pathway likewise uses a retrieval-oriented prompt that compresses a query into key sources, actions, and acoustic cues. This means that AuroLA does not merely encode the raw modalities; it first converts them into prompted, contextualized representations before pooling through the special retrieval token. This suggests that the retrieval space inherits part of the MLLM’s instruction-following and summarization behavior (Xu et al., 20 Feb 2026).

4. Hybrid-NCE and hard-negative reweighting

The principal training objective is a Hybrid-NCE loss that combines multi-granular supervision with hard-negative reweighting (Xu et al., 20 Feb 2026). For a granularity $k$0, with batch size $k$1, cosine similarity $k$2, and temperature $k$3, the single-granularity loss is

$k$4

Hard negatives are identified within the batch by high similarity among non-matching pairs. The weighting scheme is described as

$k$5

with an annealing schedule

$k$6

This schedule starts with uniform negatives and increases hardness gradually to avoid instability (Xu et al., 20 Feb 2026).

The full hybrid objective is

$k$7

where $k$8 spans $k$9. The description also notes $[\mathrm{AUD\text{-}EMB}]$0-normalization of embeddings, optional supervised contrastive regularization when multiple captions per audio are available, and the possibility of fixed or learned temperatures, including separate $[\mathrm{AUD\text{-}EMB}]$1 per granularity (Xu et al., 20 Feb 2026).

The significance of Hybrid-NCE is that it trains against diverse textual supervision without forcing all supervision into a single caption form. Long descriptions favor rich semantics, tags provide concise discriminative cues, and event segments contribute temporal specificity. Hard-negative reweighting then shifts learning pressure toward semantically confusing non-matches rather than easy negatives.

5. Bidirectional MLLM-based re-ranking

After initial retrieval in the shared embedding space, AuroLA performs bidirectional re-ranking through deeper cross-modal interaction in the same MLLM backbone (Xu et al., 20 Feb 2026). In the audio-to-text direction, the query audio tokens and a candidate text are combined under an instruction prompt, and cross-attention is allowed between the two streams. A scalar compatibility score is then obtained from a $[\mathrm{AUD\text{-}EMB}]$2 or $[\mathrm{AUD\text{-}EMB}]$3 token head: $[\mathrm{AUD\text{-}EMB}]$4 The text-to-audio direction is processed analogously: $[\mathrm{AUD\text{-}EMB}]$5

These scores are aggregated with the base embedding similarity $[\mathrm{AUD\text{-}EMB}]$6 according to

$[\mathrm{AUD\text{-}EMB}]$7

where $[\mathrm{AUD\text{-}EMB}]$8 controls the balance between fast embedding retrieval and slower but more accurate cross-modal re-ranking (Xu et al., 20 Feb 2026). Re-ranking is run only on top-$[\mathrm{AUD\text{-}EMB}]$9 candidates, with the description giving $[\mathrm{TXT\text{-}EMB}]$0–$[\mathrm{TXT\text{-}EMB}]$1 as the relevant compute-control regime.

This module is designed for cases where shallow embedding similarity is insufficient. Sparse tags, incomplete captions, or acoustically similar candidates can remain ambiguous after first-stage retrieval. The re-ranker uses the MLLM’s cross-attention to inspect detailed interactions between the query and each candidate, which the overview describes as especially useful for disambiguation in the top-$[\mathrm{TXT\text{-}EMB}]$2 set (Xu et al., 20 Feb 2026).

6. Evaluation, scaling behavior, and limitations

The reported empirical result is that AuroLA consistently outperforms state-of-the-art models, including the recent PE-AV, while utilizing only approximately $[\mathrm{TXT\text{-}EMB}]$3 of PE-AV’s training data (Xu et al., 20 Feb 2026). The framework is also described as showing clear scaling trends with respect to dataset size and model capacity, which is presented as validation of the MLLM as a unified backbone for audio–text retrieval (Xu et al., 20 Feb 2026).

The evaluation regime is described in conventional audio–text retrieval terms. Benchmarks referenced in the overview include AudioCaps retrieval, Clotho retrieval, and VGGSound retrieval, while metrics include Recall@$[\mathrm{TXT\text{-}EMB}]$4, mean Average Precision, median rank, and segment-level mAP for long-audio or segment retrieval. However, exact quantitative tables, fitted scaling laws, and detailed ablation numbers are not specified in the description. The same applies to exact backbone choice, training steps, batch sizes, and total compute (Xu et al., 20 Feb 2026).

The overview also delineates several limitations. Data biases can arise because library distributions overrepresent certain sound effects, and automated pipelines can propagate weak or noisy labels. The framework is described as focusing on non-speech ambient and event audio, so dedicated speech or ASR integration is identified as a possible extension. Re-ranking adds latency, long-audio chunking may miss global narratives, and multilingual retrieval remains an open direction. This suggests that AuroLA’s main contribution is architectural and training-oriented rather than a complete solution to all audio–language alignment regimes (Xu et al., 20 Feb 2026).

A common misconception is to treat AuroLA as merely a larger dual encoder. The available description indicates a broader design: a scalable annotation pipeline, prompted MLLM summarization, special-token embedding extraction, a multi-granular Hybrid-NCE objective, and bidirectional re-ranking all function as integral parts of the system. Another misconception is that the MLLM alone explains the gains. The framework as described couples MLLM capacity with supervision design and hard-negative handling, implying that retrieval quality emerges from the interaction of backbone, data curation, and training objective rather than from scale alone (Xu et al., 20 Feb 2026).