Contrastive Audio-Vision Fusion

• Contrastive learning-based audio-vision fusion is a multimodal approach that aligns audio and visual representations with contrastive objectives to emphasize semantically matching pairs.
• It leverages loss functions such as InfoNCE in various architectures, enhancing performance in tasks like speech enhancement, action recognition, and scene classification.
• Empirical evaluations show that when contrastive signals and pairing strategies are well-tuned, they yield significant improvements in reconstruction fidelity and classification accuracy.

Searching arXiv for the focal paper and closely related work on contrastive audio-visual fusion. Contrastive learning-based audio-vision fusion denotes a family of multimodal learning methods in which audio and visual representations are brought into a shared or coordinated latent structure by objectives that reward agreement for matched cross-modal observations and penalize mismatch for non-corresponding ones. In the literature, this paradigm appears in self-supervised representation learning, weakly supervised parsing, action recognition, scene classification, segmentation, conditional generation, diffusion-based speech enhancement, retrieval, and phonological analysis. The common technical premise is that audio-visual fusion is not only a matter of concatenating features or applying attention, but also of shaping the geometry of the learned representation space so that temporally, semantically, or spatially corresponding signals become easier to fuse downstream (Morgado et al., 2020).

1. Conceptual scope and problem formulation

In multimodal pipelines, a recurring problem is that high similarity within a modality does not imply reliable semantic correspondence across modalities. Wu et al. explicitly measured this discrepancy on the LLP dataset: when the highest-cosine-similarity pair is selected within a modality, label agreement is frequent, whereas across audio and vision the top-similarity snippets agree in label only about one third of the time. They characterize this as a misalignment between feature similarity and feature semantic similarity, and introduce multimodal time-series contrastive loss to correct it (Wu et al., 2021).

This concern also appears in later audio-visual systems that are structurally very different. In self-supervised video representation learning, cross-modal discrimination is reported to be more effective than within-modal discrimination for learning useful audio-visual features, and cross-modal agreement is used to refine the definition of positives and negatives beyond single-instance matching (Morgado et al., 2020). In scene classification, fine-grained event-object alignment is treated as a prerequisite for reliable semantic fusion because clip-level fusion ignores detailed audio events and visual objects (Hou et al., 2022). In diffusion-based audio-visual speech enhancement, the question is whether visual conditioning injected by cross-attention is actually used by the denoiser; the answer is addressed by adding a contrastive audio-visual loss to the diffusion objective while leaving the posterior-sampling inference framework unchanged (Mboungou et al., 16 Jun 2026).

Across these formulations, “fusion” therefore refers to more than late combination. It includes latent-space coordination, cross-attention conditioning, composition of modality-specific and multimodal embeddings, and auxiliary contrastive supervision that changes how one modality constrains the other. A plausible implication is that contrastive learning-based fusion is best viewed as a representational regularization strategy that can be attached to many downstream architectures rather than as a single fusion block.

2. Contrastive objectives and alignment criteria

The dominant loss family is InfoNCE or closely related noise-contrastive objectives. In the diffusion-based visual-conditioned speech enhancement model, the generative score-matching loss

$$L_{\mathrm{Gen}} = \mathbb E_{t\sim \mathrm{Unif}(0,1),\, s_0\sim p_{\mathrm{data}},\, v,\, \zeta\sim\mathcal N(0,I)} \left[\left\|\sigma(t) S_\theta(s_t,t,v)+\zeta\right\|_2^2\right]$$

is augmented by a symmetric InfoNCE term over audio and visual embeddings extracted from Tweedie’s estimate and frozen AV-HuBERT features, producing

$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$

with $\alpha(t)=1$ for $t\le 0.3$ and $0$ otherwise, and a linear warm-up of $\beta(\mathrm{epoch})$ over the first 100 epochs (Mboungou et al., 16 Jun 2026).

Other works use standard bidirectional cross-modal InfoNCE. AVID defines Cross-AVID by discriminating video from audio memory vectors and audio from video memory vectors, and then extends instance discrimination with Cross-Modal Agreement, where the positive set is formed by the top-$K$ neighbors under

$\rho_{ij}=\min(v_i^\top v_j,\; a_i^\top a_j),$

so that a pair is considered strongly positive only if it is similar in both feature spaces (Morgado et al., 2020). LAVA uses pairwise audio-video and video-text NCE together with a centroid loss over audio, video, and text to enforce a tri-modal manifold (Gurram et al., 2022). SLAVA uses pairwise $L_{AV}$, $L_{AT}$, and $$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$0 in a single-stage trimodal setting, explicitly comparing indirect alignment through text with direct inclusion of an audio-visual loss (Sudarsanam et al., 20 May 2025).

Some formulations depart from vanilla InfoNCE while remaining contrastive in spirit. Wu et al. propose a time-aware multimodal time-series contrastive loss in which the time-lag coefficient is

$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$1

and the regression-style objective uses detached unimodal similarity to regulate cross-modal similarity across time indices (Wu et al., 2021). AVCL combines a co-correlation guided representation alignment loss, which pushes the cross-correlation matrix toward the identity, with a cross-modal contrastive loss over audio-visual positives and intra-/inter-modal negatives (Liu et al., 2022). In phonological class recognition, audio and rtMRI embeddings are aligned with a cosine-embedding term

$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$2

combined with class-weighted cross-entropy (Liu et al., 23 Jul 2025). In audio-visual scene classification, contrast is imposed not on whole clips but on the top-$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$3 and bottom-$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$4 classifier weight vectors representing audio events and visual objects (Hou et al., 2022).

In diffusion-based conditional generation and segmentation, contrastive learning can be expressed through denoising or density-ratio objectives rather than only through embedding-space classification. CMMD subtracts denoising losses on mismatched audio or video negatives from the positive-pair joint diffusion loss, with weighting factor $$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$5 (Yang et al., 2023). The audio-visual segmentation diffusion model optimizes a latent diffusion objective together with an InfoNCE-style term that approximates maximizing the log-density ratio between multimodal and audio-only conditioning (Mao et al., 2023).

3. Fusion architectures and representation pathways

Architecturally, contrastive audio-vision fusion spans dual encoders, shared encoders, cross-attention U-Nets, transformers, and latent-diffusion models. The diffusion-based AVSE system centers on a conditional score-based diffusion network $$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$6 implemented as a U-Net–style variant of NCSN++ (“NCSN++M”) with approximately $$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$7 million parameters. At each resolution stage, projected AV-HuBERT visual features are injected into the audio pathway through single-head cross-attention:

$$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$8

with the result added back into the U-Net activations (Mboungou et al., 16 Jun 2026).

Encoder-only dual-stream designs remain common. AVID uses an R(2+1)D video encoder, a 2D ConvNet audio encoder, projection MLPs, and a memory bank storing exponential moving averages of past embeddings (Morgado et al., 2020). LAVA uses transformer encoders for audio, video, and text, with linear projection functions into shared multimodal spaces (Gurram et al., 2022). The music-video study also adopts a dual-encoder structure, but freezes musicnn and R(2+1)D backbones and trains only small projection heads, a design choice later implicated in its negative results (Veldkamp et al., 2023).

Several models combine contrastive alignment with explicit fusion modules. AVCL introduces an attention-based multi-modal fusion module in which concatenated audio and visual features are projected to a joint embedding, transformed into an excitation vector, and then used for channel-wise gating of both modalities before contrastive alignment and co-correlation regularization (Liu et al., 2022). In audio-visual scene classification, contrastive event-object alignment is followed by semantic-based fusion implemented with cross-attention between event logits and object logits, producing enriched embeddings that are concatenated for scene prediction (Hou et al., 2022). In weakly supervised audio-visual video parsing, contrastive alignment is performed before the Hybrid Attention Network consumes the aligned snippet features (Wu et al., 2021).

Masked-modelling systems incorporate a separate fused stream. CAV-MAE processes audio and visual patches with per-modality ViT-style encoders, then passes unmasked tokens into a three-stream joint encoder: audio-only, visual-only, and audio+visual. The modality-pure streams produce contrastive embeddings, while the fused stream is used for masked reconstruction and downstream classification (Gong et al., 2022). By contrast, FSSUAVL avoids explicit cross-modal pairing altogether: audio is converted to spectrograms so that a single backbone encoder and projection head can process both images and audio in a common embedding space, and local federated batches contain only one modality at a time (Rehman et al., 13 Apr 2025).

Diffusion-based generative models add another form of fusion. The audio-visual segmentation model encodes visual and audio features into a joint latent condition $$L = L_{\mathrm{Gen}} + \alpha(t)\cdot \beta(\mathrm{epoch})\cdot L_{\mathrm{InfoNCE}},$$9 that enters the denoising U-Net, while the segmentation latent $\alpha(t)=1$0 provides the ground-truth latent target during training (Mao et al., 2023). CMMD proposes an “easy fusion” block that temporally resamples video features, spatially pools them, replicates audio features across the reduced grid, concatenates the two tensors channel-wise, and then relies on U-Net self-attention to realize cross-modal interaction more cheaply than standard cross-attention (Yang et al., 2023).

4. Application domains

A notable characteristic of the field is the breadth of tasks to which contrastive audio-vision fusion has been applied.

Application area	Representative formulation	Representative paper
Speech enhancement	Diffusion prior with visual cross-attention and scheduled InfoNCE	(Mboungou et al., 16 Jun 2026)
Video representation learning	Cross-modal instance discrimination and agreement-based positives	(Morgado et al., 2020)
Weakly supervised parsing	Time-aware snippet alignment before Hybrid Attention fusion	(Wu et al., 2021)
Action recognition	Attention fusion plus co-correlation and self-supervised contrastive learning	(Liu et al., 2022)
Scene classification	Event-object alignment with semantic cross-attention fusion	(Hou et al., 2022)
Segmentation and generation	Contrastive latent diffusion and joint denoising constraints	(Mao et al., 2023, Yang et al., 2023)

In speech enhancement, the contrastive term is used to strengthen the use of lip movements without altering the EM-style posterior sampling inference algorithm inherited from AV-UDiffSE+ (Mboungou et al., 16 Jun 2026). In phonological analysis, synchronized speech and rtMRI are aligned during training, but inference uses the ViT branch alone; this is a distinctive “training-only multimodal supervision” configuration (Liu et al., 23 Jul 2025). In self-supervised pretraining, the learned audio-visual representation is later transferred to action recognition, audio classification, retrieval, or segmentation tasks (Morgado et al., 2020, Gong et al., 2022, Morgado et al., 2020).

The topic also extends beyond paired supervised settings. FSSUAVL studies federated self-supervised training on decentralized and potentially unpaired image and audio data, with no explicit cross-modal positives and no need for auxiliary pretrained encoders or generative models on clients (Rehman et al., 13 Apr 2025). This suggests a broader interpretation of contrastive audio-vision fusion in which a single shared embedding space can function as a modality-agnostic discriminative substrate even when direct cross-modal correspondence is unavailable.

Trimodal extensions are also part of the landscape. LAVA and SLAVA both include text as an additional anchor or mediator for audio and video alignment (Gurram et al., 2022, Sudarsanam et al., 20 May 2025). In these models, audio-vision fusion is embedded within a larger multimodal geometry rather than learned in isolation.

5. Empirical patterns, gains, and ablations

The empirical literature repeatedly reports that explicit cross-modal alignment improves downstream fusion, but the magnitude and reliability of the gain depend strongly on the task and on how positives and negatives are defined.

In diffusion-based visual-conditioned speech enhancement, augmenting the diffusion score-matching loss with scheduled InfoNCE yields consistent gains over the AV-DiffUSEEN baseline across matched and mismatched conditions. On matched TCD-DEMAND, the method reports an overall SI-SIR gain of $\alpha(t)=1$1 dB and an SI-SDR gain of $\alpha(t)=1$2 dB; at $\alpha(t)=1$3 dB SNR, SI-SIR improves from $\alpha(t)=1$4 to $\alpha(t)=1$5 dB and SI-SDR from $\alpha(t)=1$6 to $\alpha(t)=1$7 dB. On mismatched LRS3-NTCD, the overall gains are $\alpha(t)=1$8 dB SI-SIR, $\alpha(t)=1$9 dB SI-SDR, and $t\le 0.3$0 PESQ (Mboungou et al., 16 Jun 2026). The same paper also reports that masking the visual input at inference causes a much larger SI-SDR drop than in the baseline, which is presented as evidence that the model has learned stronger reliance on visual cues.

In weakly supervised audio-visual video parsing, full MTSC improves the segment-level Type@AV F-score from $t\le 0.3$1 to $t\le 0.3$2 and event-level from $t\le 0.3$3 to $t\le 0.3$4, while best-backbone selection alone yields smaller gains (Wu et al., 2021). In action recognition, AVCL reaches $t\le 0.3$5 top-1 on Kinetics-Sounds32 and $t\le 0.3$6 on Kinetics-Sounds100 with an R(2+1)D backbone, and ablations attribute additional value to both AMFM and CGRA (Liu et al., 2022). In audio-visual scene classification, adding contrastive event-object alignment and semantic-based fusion raises accuracy from $t\le 0.3$7 for the backbone-only system to $t\le 0.3$8, and the full audio-visual model reaches $t\le 0.3$9 compared with $0$0 for audio-only and $0$1 for visual-only (Hou et al., 2022).

Self-supervised pretraining studies report analogous patterns. AVID shows Cross-AVID outperforming Self-AVID, with AVID+CMA improving linear-probe and downstream action recognition results over Cross-AVID alone (Morgado et al., 2020). CAV-MAE reports that adding contrastive correspondence to masked reconstruction yields better joint representations than either component alone, with $0$2 mAP on AudioSet-20K for CAV-MAE versus $0$3 for contrastive-only CAV and $0$4 for reconstruction-only AV-MAE; on VGGSound, CAV-MAE reaches $0$5 and its Scale+ variant $0$6 (Gong et al., 2022). In 360° spatial alignment, AVSA-pretrained features outperform AVC- and AVTS-pretrained ones on correspondence, semantic segmentation, and action recognition, including $0$7 on UCF101 and $0$8 on HMDB51 (Morgado et al., 2020).

Phonological classification provides a domain-specific example with a large absolute improvement: the contrastive audio-vision system on USC-TIMIT reaches an average F1-score of $0$9, compared with $\beta(\mathrm{epoch})$0 for middle fusion, $\beta(\mathrm{epoch})$1 for unimodal audio, and $\beta(\mathrm{epoch})$2 for unimodal vision (Liu et al., 23 Jul 2025). In trimodal retrieval, SLAVA reports that single-stage training improves visual retrieval from raw audio from about $\beta(\mathrm{epoch})$3 in the two-stage baseline to $\beta(\mathrm{epoch})$4 in SLAVA_A V, while SLAVA_AV with $\beta(\mathrm{epoch})$5 reaches $\beta(\mathrm{epoch})$6 (Sudarsanam et al., 20 May 2025).

A recurring ablation result is that the weight on the contrastive term must be carefully tuned. In AVSE, too small a $\beta(\mathrm{epoch})$7 gives no benefit and too large a value harms reconstruction, with the best trade-off reported at $\beta(\mathrm{epoch})$8 (Mboungou et al., 16 Jun 2026). In CAV-MAE, the best contrastive coefficient is $\beta(\mathrm{epoch})$9, with larger values causing the contrastive component to dominate and smaller values weakening alignment (Gong et al., 2022). In scene classification, lowest-$K$0 hard-negative sampling slightly outperforms random-$K$1 sampling (Hou et al., 2022).

6. Limitations, failure modes, and points of contention

Despite many positive results, the literature does not support a universal claim that contrastive audio-vision fusion is always effective. The clearest counterexample is the music-video domain, where a straightforward bidirectional cross-modal InfoNCE objective trained on more than half a million music videos fails to improve over frozen pretrained backbones. Median retrieval ranks remain around $K$2 in a batch of $K$3, and downstream AUC drops relative to musicnn or musicnn+R(2+1)D baselines (Veldkamp et al., 2023). The paper attributes this to a “heterogeneity gap”: music videos often encode mood, narrative, or visual style rather than concrete audio events, so the two modalities are only loosely related.

This negative result clarifies a broader misconception. Contrastive fusion does not create semantic correspondence where the data distribution provides little or no systematic cross-modal signal. The success of the method depends on the structure of positive pairs. In action-centric video, speech reading, scene understanding, and spatial audio tasks, same-instance or same-time pairing often carries useful semantics. In music video, the pairing may be too weak or too abstract for instance-level InfoNCE to capture (Veldkamp et al., 2023).

Another recurring issue is collapse or over-reliance on simplistic similarity structure. AVID reports that Cross-AVID alone produces an average random-pair dot product of about $K$4, and CMA is introduced partly to restore the expected average of $K$5 (Morgado et al., 2020). Wu et al. argue that naïvely assuming “feature-similarity $K$6 semantic-similarity” causes attention or transformer fusion layers to attend to semantically unrelated signals (Wu et al., 2021). The 360° spatial alignment work similarly argues that clip-level correspondence is insufficient because it ignores sound-source direction and can spuriously associate non-sounding co-occurring objects with the soundtrack (Morgado et al., 2020).

Generative and diffusion-based models introduce an additional concern: the conditioning modality may be architecturally present but functionally weak. Several diffusion papers therefore add contrastive or density-ratio terms specifically to guarantee that the conditional variable contributes to the output (Mboungou et al., 16 Jun 2026, Mao et al., 2023, Yang et al., 2023). A plausible implication is that, in iterative generative models, contrastive objectives often serve less as standalone representation learners than as diagnostics and remedies for conditional under-utilization.

7. Research directions and synthesis

The literature suggests several converging directions. One is the move from pairwise alignment to structured alignment: time-aware weighting in MTSC, cross-modal agreement sets in AVID, event-object alignment in scene classification, and spatially localized positives and negatives in AVSA all refine the notion of what constitutes a positive pair (Wu et al., 2021, Morgado et al., 2020, Hou et al., 2022, Morgado et al., 2020). Another is integration with generative modeling. Diffusion-based speech enhancement, audio-visual segmentation, and bi-directional video-audio generation each use contrastive learning not as the primary generator, but as a constraint that increases the informativeness of the conditioning signal (Mboungou et al., 16 Jun 2026, Mao et al., 2023, Yang et al., 2023).

A second trend is the expansion from bimodal to trimodal settings. LAVA uses audio-video and video-text pairwise losses plus a centroid tri-modal loss (Gurram et al., 2022). SLAVA explicitly compares indirect alignment through text against direct inclusion of $K$7 and reports that adding explicit audio-visual contrastive loss improves audio-based visual retrieval beyond indirect audio-text and visual-text alignment alone (Sudarsanam et al., 20 May 2025). This suggests that text can function either as an auxiliary anchor or as a partial but insufficient surrogate for direct audio-vision fusion.

A third direction concerns the role of supervision and deployment constraints. Some methods use contrastive fusion only during training and deploy a single modality at test time, as in phonological class recognition (Liu et al., 23 Jul 2025). Others preserve the same inference-time algorithm and change only the training loss, as in diffusion-based AVSE (Mboungou et al., 16 Jun 2026). Federated approaches demonstrate that a shared multimodal embedding can also be learned without paired cross-modal positives, though this shifts the objective from alignment to joint discrimination (Rehman et al., 13 Apr 2025).

Taken together, these results support a technically narrow but robust conclusion: contrastive learning-based audio-vision fusion is most effective when the contrastive signal is matched to the structure of cross-modal correspondence in the data—temporal, spatial, semantic, or conditional—and when the alignment objective is integrated with, rather than substituted for, the downstream fusion mechanism. Where this condition holds, the method consistently improves interference suppression, reconstruction fidelity, retrieval, classification, parsing, segmentation, or representation transfer; where it does not, contrastive alignment can remain superficial or fail outright (Mboungou et al., 16 Jun 2026, Veldkamp et al., 2023).