Hybrid-NCE Loss for Audio-Text Retrieval

• Hybrid-NCE Loss is a hybrid objective that combines symmetric InfoNCE, cosine similarity, and L1 losses to align audio–text embeddings.
• The formulation augments pure contrastive learning with positive-only terms, reducing gradient variance and countering batch size limitations.
• Empirical results on datasets like Clotho and AudioCaps demonstrate improved retrieval performance and noise robustness compared to baselines.

“Hybrid-NCE loss” is an Editor’s term for the loss formulation described in "Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss" (Liu et al., 25 Apr 2026). The paper itself does not introduce the label “Hybrid-NCE”; it consistently describes a hybrid loss combining cosine similarity, $\mathcal{L}_1$, and contrastive objectives for audio–text retrieval. In that formulation, a symmetric InfoNCE-style contrastive term with in-batch negatives is augmented by two positive-pair alignment terms: a cosine “directional” loss and an $\mathcal{L}_1$ loss. The loss is designed for multimodal retrieval under long-form, noisy, and weakly labeled audio, and is coupled to a training regime in which cross-modal attention refines projection parameters during training while retrieval is performed with independent per-modality embeddings at inference time (Liu et al., 25 Apr 2026).

1. Conceptual definition and scope

The hybrid loss operates on a minibatch of $N$ matched audio–text pairs $\{(x_i^a, x_i^t)\}_{i=1}^N$, with one shared-space audio embedding $a_i$ and one shared-space text embedding $t_i$ produced per pair. Its defining characteristic is the combination of three objectives with distinct roles: relative ranking through symmetric InfoNCE, directional alignment through cosine similarity, and elementwise proximity through $\mathcal{L}_1$ distance.

Within the terminology of the paper, the contrastive component is a straight InfoNCE objective, not NCE with an explicit noise distribution. It is symmetrized over both retrieval directions, audio-to-text and text-to-audio, and uses standard in-batch normalization with no reported memory bank, no hard-negative mining, and no EMA memory. The additional cosine and $\mathcal{L}_1$ terms are positive-only constraints defined on matched pairs. This structure makes the formulation an augmentation of standard dual-encoder retrieval losses rather than a replacement for InfoNCE itself (Liu et al., 25 Apr 2026).

A central design choice is that the loss is computed on the same independent, per-modality embeddings that are used at inference. Cross-modal attention is present during training, but only as a refinement mechanism for the projection stack. This avoids a train–test mismatch in which the retrieval score would depend on interactions unavailable at inference.

2. Mathematical formulation

Before similarity is computed, the embeddings are $\ell_2$-normalized:

$$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$

The similarity function is cosine similarity:

$\mathcal{L}_1$0

The contrastive component is symmetric InfoNCE with temperature $\mathcal{L}_1$1:

$\mathcal{L}_1$2

$\mathcal{L}_1$3

$\mathcal{L}_1$4

The cosine “directional” term is

$\mathcal{L}_1$5

The $\mathcal{L}_1$6 term is

$\mathcal{L}_1$7

The total loss is a convex combination:

$\mathcal{L}_1$8

The paper reports ablations on Clotho with HTSAT-tiny and RoBERTa-large, with frozen encoders, and identifies $\mathcal{L}_1$9 as the best validation setting. A pure-contrastive baseline corresponds to $N$0. The temperature $N$1 is not specified in the paper; it is treated as a hyperparameter, with a suggested search range $N$2 for reproduction when no further guidance is available (Liu et al., 25 Apr 2026).

3. Embedding pathway and placement of the loss

The embeddings used in the loss are the outputs of a projection and refinement stack applied to encoder features. The audio encoder yields frame-level or chunk-level vectors, and the text encoder yields token-level vectors. For each modality, the refinement stack has three stages: a transformer-based projection, a linear mapping into the shared space, and a cross-modal attention stage used during training only.

For an input sequence $N$3, the transformer-based projection consists of one Transformer encoder block with multi-head attention using 8 heads, a GELU feed-forward network with hidden size $N$4, dropout $N$5, and residual connections:

$N$6

The linear mapping into the shared space is

$N$7

where $N$8 and $N$9.

During training, cross-modal attention is defined for shared-space sequences $\{(x_i^a, x_i^t)\}_{i=1}^N$0 and $\{(x_i^a, x_i^t)\}_{i=1}^N$1 by

$\{(x_i^a, x_i^t)\}_{i=1}^N$2

$\{(x_i^a, x_i^t)\}_{i=1}^N$3

with a symmetric text-to-audio path. At inference, cross-modal attention is disabled, so only the per-modality transformer projection and linear mapping remain. The paper explicitly states that this preserves dual-encoder efficiency (Liu et al., 25 Apr 2026).

The loss is applied after pooling and projection, on one audio embedding $\{(x_i^a, x_i^t)\}_{i=1}^N$4 and one text embedding $\{(x_i^a, x_i^t)\}_{i=1}^N$5 per pair, both in $\{(x_i^a, x_i^t)\}_{i=1}^N$6. The pooled audio vector and pooled text vector are the inputs to the projection module that yields these embeddings. Although cross-modal attention participates in training through backpropagation, the similarity function $\{(x_i^a, x_i^t)\}_{i=1}^N$7, the directional term, and the $\{(x_i^a, x_i^t)\}_{i=1}^N$8 term are all computed on the independent per-modality outputs. This design is explicitly intended to avoid train–test mismatch.

4. Small-batch behavior and robustness mechanism

The paper motivates the hybridization by identifying failure modes of pure InfoNCE under small-batch constraints. With $\{(x_i^a, x_i^t)\}_{i=1}^N$9 pairs per batch, each anchor has only $a_i$0 negatives. According to the paper, fewer negatives reduce the contrastive signal and increase gradient variance, while mislabeled negatives arising from weak labels can produce biased gradients that push semantically related items apart. It also notes that with small $a_i$1 and small $a_i$2, the softmax may become peaky and the loss can saturate early, leaving little gradient when positives already dominate the denominator (Liu et al., 25 Apr 2026).

The two positive-only terms are introduced precisely to counter those effects. The directional loss and the $a_i$3 loss do not depend on negatives, so they provide consistent, low-variance gradients that pull matched pairs together even when the contrastive signal is weak or noisy. The paper characterizes them as regularizers that anchor absolute alignment in the shared space while InfoNCE supplies the relative ranking signal.

The stated gradient intuition is explicit. Defining

$a_i$4

the audio-to-text InfoNCE gradients satisfy

$a_i$5

and, for $a_i$6,

$a_i$7

The paper further states that the directional term contributes

$a_i$8

which yields a constant attractive force on positives after the chain rule through cosine similarity and $a_i$9-normalization. The $t_i$0 term adds an attractive subgradient on each embedding dimension by reducing $t_i$1 and is described as less sensitive than $t_i$2 to occasional large errors. Increasing $t_i$3 smooths $t_i$4 and reduces the magnitude of the InfoNCE gradients, but the paper states that $t_i$5 tuning alone cannot compensate for a lack of negatives.

5. Long-form audio, noisy conditions, and pooling

The hybrid loss is integrated with a long-audio pipeline based on silence-aware chunking and attention-based pooling. Positive pairs are formed by each audio clip and its paired caption or captions. The paper does not restrict multiple captions per audio; it states that each caption can form a separate positive with the same audio embedding as typical practice. Negatives remain standard in-batch negatives in both retrieval directions.

The preprocessing rule for long audio is explicit: remove silences longer than 1 second and then segment the remaining signal into fixed 10-second chunks. Each chunk is encoded into an audio feature vector. During training, chunk-level features are pooled by attention using the paired text embedding as query:

$t_i$6

The paper states that this routing focuses the audio embedding on segments relevant to the caption and implicitly downweights silent or noisy parts. At inference, the paired text embedding is unavailable, so a learned query $t_i$7 replaces $t_i$8 to allow precomputation of audio embeddings. An additional training trick is reported: with probability $t_i$9, the text-conditioned query is replaced by the learned $\mathcal{L}_1$0 during training to reduce train–test mismatch (Liu et al., 25 Apr 2026).

The loss itself remains clip-level. The paper explicitly states that it does not include a chunk-level contrastive objective or explicit silence masking in the loss. Suppression of irrelevant segments comes instead from silence removal and attention pooling. This suggests that the hybrid loss is intended to operate on globally pooled semantic representations rather than local alignment units.

6. Empirical performance, ablations, and limitations

The paper reports consistent retrieval gains over Microsoft-CLAP and LAION-CLAP across several datasets. For audio-to-text retrieval, the reported results are: AudioCaps, $\mathcal{L}_1$1 and $\mathcal{L}_1$2; Clotho, $\mathcal{L}_1$3 and $\mathcal{L}_1$4; ESC-50, $\mathcal{L}_1$5 and $\mathcal{L}_1$6; FSD50K, $\mathcal{L}_1$7 and $\mathcal{L}_1$8. For text-to-audio retrieval, the reported results are: AudioCaps, $\mathcal{L}_1$9 and $\mathcal{L}_1$0; Clotho, $\mathcal{L}_1$1 and $\mathcal{L}_1$2. The paper also reports paired Wilcoxon signed-rank tests on per-audio $\mathcal{L}_1$3, with $\mathcal{L}_1$4 for audio-to-text and $\mathcal{L}_1$5 for text-to-audio against the second-best baseline (Liu et al., 25 Apr 2026).

Ablations isolate the contribution of the hybrid loss on Clotho with HTSAT-tiny and RoBERTa-large, using frozen encoders. The pure contrastive setting $\mathcal{L}_1$6 yields $\mathcal{L}_1$7 for audio-to-text and $\mathcal{L}_1$8 for text-to-audio, whereas the hybrid setting $\mathcal{L}_1$9 yields $\ell_2$0 and $\ell_2$1, corresponding to gains of $\ell_2$2 and $\ell_2$3. The appendix further reports that replacing a linear projection with a Transformer projection and using the combined loss consistently yields the strongest performance across encoder choices.

Noise robustness is evaluated under additive MUSAN noise with SNR from 5 to 15. At SNR 10 for audio-to-text $\ell_2$4, the reported comparisons are: Clotho, $\ell_2$5 versus $\ell_2$6 for Microsoft-CLAP and $\ell_2$7 for LAION-CLAP; AudioCaps, $\ell_2$8 versus $\ell_2$9 and $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$0; ESC-50, $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$1 versus $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$2 and $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$3; FSD50K, $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$4 versus $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$5 and $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$6. The paper attributes these improvements to silence-aware chunking, attention pooling, and the hybrid loss’s reduced reliance on large negative sets.

Sensitivity to batch size is reported for Clotho with HTSAT-tiny and RoBERTa-large: batch size 4 gives audio-to-text and text-to-audio $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$7 of $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$8 and $$\hat{a}_i = \frac{a_i}{\|a_i\|_2}, \qquad \hat{t}_i = \frac{t_i}{\|t_i\|_2}.$$9; batch size 8 gives $\mathcal{L}_1$00 and $\mathcal{L}_1$01; batch size 16 gives $\mathcal{L}_1$02 and $\mathcal{L}_1$03; batch size 32 gives $\mathcal{L}_1$04 and $\mathcal{L}_1$05; batch size 64 gives $\mathcal{L}_1$06 and $\mathcal{L}_1$07. The paper characterizes the model as relatively robust and states that the hybrid loss and pooling appear to mitigate the typical small-batch degradation of pure InfoNCE.

The limitations listed are specific. Performance depends on strong pretrained encoders, and smaller or weaker encoders may reduce gains. Extremely noisy, polyphonic scenes and overlapping events remain challenging because attention pooling may miss weak target events. Silence-based segmentation is characterized as a coarse heuristic, and difficult backgrounds may require more adaptive segmentation. For transfer to other multimodal retrieval tasks, the paper advises keeping the same $\mathcal{L}_1$08 on inference-time embeddings, reserving cross-attention for training-only refinement, and increasing $\mathcal{L}_1$09 or $\mathcal{L}_1$10 modestly when batches are very small or labels are noisy while retaining $\mathcal{L}_1$11 when moderate batch sizes are available.