---
title: Contrastive Audio-Vision Fusion
url: https://www.emergentmind.com/topics/contrastive-learning-based-audio-vision-fusion
type: topic
---

# Contrastive Audio-Vision Fusion

Searching arXiv for the focal paper and closely related work on contrastive audio-visual fusion.
Contrastive learning-based audio-vision fusion denotes a family of multimodal learning methods in which audio and visual representations are brought into a shared or coordinated latent structure by objectives that reward agreement for matched cross-modal observations and penalize mismatch for non-corresponding ones. In the literature, this paradigm appears in self-supervised representation learning, weakly supervised parsing, action recognition, scene classification, segmentation, conditional generation, diffusion-based speech enhancement, retrieval, and phonological analysis. The common technical premise is that audio-visual fusion is not only a matter of concatenating features or applying attention, but also of shaping the geometry of the learned representation space so that temporally, semantically, or spatially corresponding signals become easier to fuse downstream [2004.12943].

## 1. Conceptual scope and problem formulation

In multimodal pipelines, a recurring problem is that high similarity within a modality does not imply reliable semantic correspondence across modalities. Wu et al. explicitly measured this discrepancy on the LLP dataset: when the highest-cosine-similarity pair is selected within a modality, label agreement is frequent, whereas across audio and vision the top-similarity snippets agree in label only about one third of the time. They characterize this as a misalignment between feature similarity and feature semantic similarity, and introduce multimodal time-series contrastive loss to correct it [2105.14430].

This concern also appears in later audio-visual systems that are structurally very different. In self-supervised video representation learning, cross-modal discrimination is reported to be more effective than within-modal discrimination for learning useful audio-visual features, and cross-modal agreement is used to refine the definition of positives and negatives beyond single-instance matching [2004.12943]. In scene classification, fine-grained event-object alignment is treated as a prerequisite for reliable semantic fusion because clip-level fusion ignores detailed audio events and visual objects [2208.02086]. In diffusion-based audio-visual speech enhancement, the question is whether visual conditioning injected by cross-attention is actually used by the denoiser; the answer is addressed by adding a contrastive audio-visual loss to the diffusion objective while leaving the posterior-sampling inference framework unchanged [2606.23712].

Across these formulations, “fusion” therefore refers to more than late combination. It includes latent-space coordination, cross-attention conditioning, composition of modality-specific and multimodal embeddings, and auxiliary contrastive supervision that changes how one modality constrains the other. A plausible implication is that contrastive learning-based fusion is best viewed as a representational regularization strategy that can be attached to many downstream architectures rather than as a single fusion block.

## 2. Contrastive objectives and alignment criteria

The dominant loss family is InfoNCE or closely related noise-contrastive objectives. In the diffusion-based visual-conditioned speech enhancement model, the generative score-matching loss
$$
L_{\mathrm{Gen}}
=
\mathbb E_{t\sim \mathrm{Unif}(0,1),\, s_0\sim p_{\mathrm{data}},\, v,\, \zeta\sim\mathcal N(0,I)}
\left[\left\|\sigma(t) S_\theta(s_t,t,v)+\zeta\right\|_2^2\right]
$$
is augmented by a symmetric InfoNCE term over audio and visual embeddings extracted from Tweedie’s estimate and frozen AV-HuBERT features, producing
$$
L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},
$$
with $\alpha(t)=1$ for $t\le 0.3$ and $0$ otherwise, and a linear warm-up of $\beta(\mathrm{epoch})$ over the first 100 epochs [2606.23712].

Other works use standard bidirectional cross-modal InfoNCE. AVID defines Cross-AVID by discriminating video from audio memory vectors and audio from video memory vectors, and then extends instance discrimination with Cross-Modal Agreement, where the positive set is formed by the top-$K$ neighbors under
$$
\rho_{ij}=\min(v_i^\top v_j,\; a_i^\top a_j),
$$
so that a pair is considered strongly positive only if it is similar in both feature spaces [2004.12943]. LAVA uses pairwise audio-video and video-text NCE together with a centroid loss over audio, video, and text to enforce a tri-modal manifold [2207.08024]. SLAVA uses pairwise $L_{AV}$, $L_{AT}$, and $L_{VT}$ in a single-stage trimodal setting, explicitly comparing indirect alignment through text with direct inclusion of an audio-visual loss [2505.14562].

Some formulations depart from vanilla InfoNCE while remaining contrastive in spirit. Wu et al. propose a time-aware multimodal time-series contrastive loss in which the time-lag coefficient is
$$
T_{i,j}=\frac{1}{|i-j|^5},
$$
and the regression-style objective uses detached unimodal similarity to regulate cross-modal similarity across time indices [2105.14430]. AVCL combines a co-correlation guided representation alignment loss, which pushes the cross-correlation matrix toward the identity, with a cross-modal contrastive loss over audio-visual positives and intra-/inter-modal negatives [2204.13386]. In phonological class recognition, audio and rtMRI embeddings are aligned with a cosine-embedding term
$$
L_{\mathrm{cos}}(x,y)=1-\cos(x,y),
$$
combined with class-weighted cross-entropy [2507.17682]. In audio-visual scene classification, contrast is imposed not on whole clips but on the top-$K$ and bottom-$K$ classifier weight vectors representing audio events and visual objects [2208.02086].

In diffusion-based conditional generation and segmentation, contrastive learning can be expressed through denoising or density-ratio objectives rather than only through embedding-space classification. CMMD subtracts denoising losses on mismatched audio or video negatives from the positive-pair joint diffusion loss, with weighting factor $\eta=5\times 10^{-5}$ [2312.05412]. The audio-visual segmentation diffusion model optimizes a latent diffusion objective together with an InfoNCE-style term that approximates maximizing the log-density ratio between multimodal and audio-only conditioning [2307.16579].

## 3. Fusion architectures and representation pathways

Architecturally, contrastive audio-vision fusion spans dual encoders, shared encoders, cross-attention U-Nets, transformers, and latent-diffusion models. The diffusion-based AVSE system centers on a conditional score-based diffusion network $S_\theta(s_t,t,v)$ implemented as a U-Net–style variant of NCSN++ (“NCSN++M”) with approximately $6.8$ million parameters. At each resolution stage, projected AV-HuBERT visual features are injected into the audio pathway through single-head cross-attention:
$$
\mathrm{Attn}(A_t,v)=\mathrm{softmax}\!\left(\frac{(A_tW_Q)(vW_K)^\top}{\sqrt d}\right)(vW_V),
$$
with the result added back into the U-Net activations [2606.23712].

Encoder-only dual-stream designs remain common. AVID uses an R(2+1)D video encoder, a 2D ConvNet audio encoder, projection MLPs, and a memory bank storing exponential moving averages of past embeddings [2004.12943]. LAVA uses transformer encoders for audio, video, and text, with linear projection functions into shared multimodal spaces [2207.08024]. The music-video study also adopts a dual-encoder structure, but freezes musicnn and R(2+1)D backbones and trains only small projection heads, a design choice later implicated in its negative results [2309.00347].

Several models combine contrastive alignment with explicit fusion modules. AVCL introduces an attention-based multi-modal fusion module in which concatenated audio and visual features are projected to a joint embedding, transformed into an excitation vector, and then used for channel-wise gating of both modalities before contrastive alignment and co-correlation regularization [2204.13386]. In audio-visual scene classification, contrastive event-object alignment is followed by semantic-based fusion implemented with cross-attention between event logits and object logits, producing enriched embeddings that are concatenated for scene prediction [2208.02086]. In weakly supervised audio-visual video parsing, contrastive alignment is performed before the Hybrid Attention Network consumes the aligned snippet features [2105.14430].

Masked-modelling systems incorporate a separate fused stream. CAV-MAE processes audio and visual patches with per-modality ViT-style encoders, then passes unmasked tokens into a three-stream joint encoder: audio-only, visual-only, and audio+visual. The modality-pure streams produce contrastive embeddings, while the fused stream is used for masked reconstruction and downstream classification [2210.07839]. By contrast, FSSUAVL avoids explicit cross-modal pairing altogether: audio is converted to spectrograms so that a single backbone encoder and projection head can process both images and audio in a common embedding space, and local federated batches contain only one modality at a time [2504.09516].

Diffusion-based generative models add another form of fusion. The audio-visual segmentation model encodes visual and audio features into a joint latent condition $c$ that enters the denoising U-Net, while the segmentation latent $z_0$ provides the ground-truth latent target during training [2307.16579]. CMMD proposes an “easy fusion” block that temporally resamples video features, spatially pools them, replicates audio features across the reduced grid, concatenates the two tensors channel-wise, and then relies on U-Net self-attention to realize cross-modal interaction more cheaply than standard cross-attention [2312.05412].

## 4. Application domains

A notable characteristic of the field is the breadth of tasks to which contrastive audio-vision fusion has been applied.

| Application area | Representative formulation | Representative paper |
|---|---|---|
| Speech enhancement | Diffusion prior with visual cross-attention and scheduled InfoNCE | [2606.23712] |
| Video representation learning | Cross-modal instance discrimination and agreement-based positives | [2004.12943] |
| Weakly supervised parsing | Time-aware snippet alignment before Hybrid Attention fusion | [2105.14430] |
| Action recognition | Attention fusion plus co-correlation and self-supervised contrastive learning | [2204.13386] |
| Scene classification | Event-object alignment with semantic cross-attention fusion | [2208.02086] |
| Segmentation and generation | Contrastive latent diffusion and joint denoising constraints | [2307.16579], [2312.05412] |

In speech enhancement, the contrastive term is used to strengthen the use of lip movements without altering the EM-style posterior sampling inference algorithm inherited from AV-UDiffSE+ [2606.23712]. In phonological analysis, synchronized speech and rtMRI are aligned during training, but inference uses the ViT branch alone; this is a distinctive “training-only multimodal supervision” configuration [2507.17682]. In self-supervised pretraining, the learned audio-visual representation is later transferred to action recognition, audio classification, retrieval, or segmentation tasks [2004.12943], [2210.07839], [2011.01819].

The topic also extends beyond paired supervised settings. FSSUAVL studies federated self-supervised training on decentralized and potentially unpaired image and audio data, with no explicit cross-modal positives and no need for auxiliary pretrained encoders or generative models on clients [2504.09516]. This suggests a broader interpretation of contrastive audio-vision fusion in which a single shared embedding space can function as a modality-agnostic discriminative substrate even when direct cross-modal correspondence is unavailable.

Trimodal extensions are also part of the landscape. LAVA and SLAVA both include text as an additional anchor or mediator for audio and video alignment [2207.08024], [2505.14562]. In these models, audio-vision fusion is embedded within a larger multimodal geometry rather than learned in isolation.

## 5. Empirical patterns, gains, and ablations

The empirical literature repeatedly reports that explicit cross-modal alignment improves downstream fusion, but the magnitude and reliability of the gain depend strongly on the task and on how positives and negatives are defined.

In diffusion-based visual-conditioned speech enhancement, augmenting the diffusion score-matching loss with scheduled InfoNCE yields consistent gains over the AV-DiffUSEEN baseline across matched and mismatched conditions. On matched TCD-DEMAND, the method reports an overall SI-SIR gain of $+5.0$ dB and an SI-SDR gain of $+2.4$ dB; at $-5$ dB SNR, SI-SIR improves from $21.2$ to $27.8$ dB and SI-SDR from $10.1$ to $13.3$ dB. On mismatched LRS3-NTCD, the overall gains are $+3.6$ dB SI-SIR, $+0.7$ dB SI-SDR, and $+0.02$ PESQ [2606.23712]. The same paper also reports that masking the visual input at inference causes a much larger SI-SDR drop than in the baseline, which is presented as evidence that the model has learned stronger reliance on visual cues.

In weakly supervised audio-visual video parsing, full MTSC improves the segment-level Type@AV F-score from $53.3$ to $55.4$ and event-level from $47.5$ to $49.1$, while best-backbone selection alone yields smaller gains [2105.14430]. In action recognition, AVCL reaches $77.2\%$ top-1 on Kinetics-Sounds32 and $67.1\%$ on Kinetics-Sounds100 with an R(2+1)D backbone, and ablations attribute additional value to both AMFM and CGRA [2204.13386]. In audio-visual scene classification, adding contrastive event-object alignment and semantic-based fusion raises accuracy from $88.42\%$ for the backbone-only system to $91.58\%$, and the full audio-visual model reaches $94.10\%$ compared with $73.55\%$ for audio-only and $88.86\%$ for visual-only [2208.02086].

Self-supervised pretraining studies report analogous patterns. AVID shows Cross-AVID outperforming Self-AVID, with AVID+CMA improving linear-probe and downstream action recognition results over Cross-AVID alone [2004.12943]. CAV-MAE reports that adding contrastive correspondence to masked reconstruction yields better joint representations than either component alone, with $40.5$ mAP on AudioSet-20K for CAV-MAE versus $38.5$ for contrastive-only CAV and $37.4$ for reconstruction-only AV-MAE; on VGGSound, CAV-MAE reaches $65.4\%$ and its Scale+ variant $65.5\%$ [2210.07839]. In 360° spatial alignment, AVSA-pretrained features outperform AVC- and AVTS-pretrained ones on correspondence, semantic segmentation, and action recognition, including $73.8\%$ on UCF101 and $37.7\%$ on HMDB51 [2011.01819].

Phonological classification provides a domain-specific example with a large absolute improvement: the contrastive audio-vision system on USC-TIMIT reaches an average F1-score of $0.81$, compared with $0.72$ for middle fusion, $0.62$ for unimodal audio, and $0.56$ for unimodal vision [2507.17682]. In trimodal retrieval, SLAVA reports that single-stage training improves visual retrieval from raw audio from about $0.27$ in the two-stage baseline to $0.52$ in SLAVA\_A V, while SLAVA\_AV with $L_{AV}+L_{AT}+L_{VT}$ reaches $0.50$ [2505.14562].

A recurring ablation result is that the weight on the contrastive term must be carefully tuned. In AVSE, too small a $\beta_0$ gives no benefit and too large a value harms reconstruction, with the best trade-off reported at $\beta_0=3000$ [2606.23712]. In CAV-MAE, the best contrastive coefficient is $\lambda_c=0.01$, with larger values causing the contrastive component to dominate and smaller values weakening alignment [2210.07839]. In scene classification, lowest-$K$ hard-negative sampling slightly outperforms random-$K$ sampling [2208.02086].

## 6. Limitations, failure modes, and points of contention

Despite many positive results, the literature does not support a universal claim that contrastive audio-vision fusion is always effective. The clearest counterexample is the music-video domain, where a straightforward bidirectional cross-modal InfoNCE objective trained on more than half a million music videos fails to improve over frozen pretrained backbones. Median retrieval ranks remain around $500$ in a batch of $1000$, and downstream AUC drops relative to musicnn or musicnn+R(2+1)D baselines [2309.00347]. The paper attributes this to a “heterogeneity gap”: music videos often encode mood, narrative, or visual style rather than concrete audio events, so the two modalities are only loosely related.

This negative result clarifies a broader misconception. Contrastive fusion does not create semantic correspondence where the data distribution provides little or no systematic cross-modal signal. The success of the method depends on the structure of positive pairs. In action-centric video, speech reading, scene understanding, and spatial audio tasks, same-instance or same-time pairing often carries useful semantics. In music video, the pairing may be too weak or too abstract for instance-level InfoNCE to capture [2309.00347].

Another recurring issue is collapse or over-reliance on simplistic similarity structure. AVID reports that Cross-AVID alone produces an average random-pair dot product of about $0.23$, and CMA is introduced partly to restore the expected average of $0$ [2004.12943]. Wu et al. argue that naïvely assuming “feature-similarity $\Rightarrow$ semantic-similarity” causes attention or transformer fusion layers to attend to semantically unrelated signals [2105.14430]. The 360° spatial alignment work similarly argues that clip-level correspondence is insufficient because it ignores sound-source direction and can spuriously associate non-sounding co-occurring objects with the soundtrack [2011.01819].

Generative and diffusion-based models introduce an additional concern: the conditioning modality may be architecturally present but functionally weak. Several diffusion papers therefore add contrastive or density-ratio terms specifically to guarantee that the conditional variable contributes to the output [2606.23712], [2307.16579], [2312.05412]. A plausible implication is that, in iterative generative models, contrastive objectives often serve less as standalone representation learners than as diagnostics and remedies for conditional under-utilization.

## 7. Research directions and synthesis

The literature suggests several converging directions. One is the move from pairwise alignment to structured alignment: time-aware weighting in MTSC, cross-modal agreement sets in AVID, event-object alignment in scene classification, and spatially localized positives and negatives in AVSA all refine the notion of what constitutes a positive pair [2105.14430], [2004.12943], [2208.02086], [2011.01819]. Another is integration with generative modeling. Diffusion-based speech enhancement, audio-visual segmentation, and bi-directional video-audio generation each use contrastive learning not as the primary generator, but as a constraint that increases the informativeness of the conditioning signal [2606.23712], [2307.16579], [2312.05412].

A second trend is the expansion from bimodal to trimodal settings. LAVA uses audio-video and video-text pairwise losses plus a centroid tri-modal loss [2207.08024]. SLAVA explicitly compares indirect alignment through text against direct inclusion of $L_{AV}$ and reports that adding explicit audio-visual contrastive loss improves audio-based visual retrieval beyond indirect audio-text and visual-text alignment alone [2505.14562]. This suggests that text can function either as an auxiliary anchor or as a partial but insufficient surrogate for direct audio-vision fusion.

A third direction concerns the role of supervision and deployment constraints. Some methods use contrastive fusion only during training and deploy a single modality at test time, as in phonological class recognition [2507.17682]. Others preserve the same inference-time algorithm and change only the training loss, as in diffusion-based AVSE [2606.23712]. Federated approaches demonstrate that a shared multimodal embedding can also be learned without paired cross-modal positives, though this shifts the objective from alignment to joint discrimination [2504.09516].

Taken together, these results support a technically narrow but robust conclusion: contrastive learning-based audio-vision fusion is most effective when the contrastive signal is matched to the structure of cross-modal correspondence in the data—temporal, spatial, semantic, or conditional—and when the alignment objective is integrated with, rather than substituted for, the downstream fusion mechanism. Where this condition holds, the method consistently improves interference suppression, reconstruction fidelity, retrieval, classification, parsing, segmentation, or representation transfer; where it does not, contrastive alignment can remain superficial or fail outright [2606.23712], [2309.00347].

Source: https://www.emergentmind.com/topics/contrastive-learning-based-audio-vision-fusion