LG-CAV-MAE: Language-Guided Audio-Visual MAE

• The paper demonstrates that LG-CAV-MAE augments masked autoencoding with language supervision, enhancing audio-visual retrieval and classification benchmarks.
• It integrates a pretrained text encoder and additional cross-modal contrastive losses to align audio, visual, and text modalities in a shared latent space.
• The method automatically generates high-quality audio-visual-text triplets using CLAP-based filtering, reducing reliance on manual annotation.

Searching arXiv for the main paper and closely related works to support citations. Language-Guided Contrastive Audio-Visual Masked Autoencoders (LG-CAV-MAE) is a tri-modal self-supervised representation-learning framework for audio-visual learning that extends Contrastive Audio-Visual Masked Autoencoder (CAV-MAE) with language supervision by integrating a pretrained text encoder and two additional cross-modal contrastive losses (Ishikawa et al., 16 Jul 2025). The method is designed to learn jointly across audio, visual, and text modalities from unlabeled videos by constructing audio-visual-text triplets automatically, rather than through manual annotation. In the reported formulation, LG-CAV-MAE combines masked reconstruction with contrastive alignment over audio-visual, audio-text, and visual-text pairs, and is evaluated on audio-visual retrieval and classification benchmarks, where it is reported to outperform prior approaches including CAV-MAE and DETECLAP (Ishikawa et al., 16 Jul 2025).

1. Position within audio-visual self-supervision

LG-CAV-MAE is explicitly presented as an extension of Contrastive Audio-Visual Masked Autoencoder (CAV-MAE), adding a pretrained text encoder and two new cross-modal contrastive losses (Ishikawa et al., 16 Jul 2025). In this sense, it occupies a specific position at the intersection of masked autoencoding, contrastive multi-modal learning, and language-guided representation learning. The framework also adopts InfoNCE, cited to Oord et al., as the contrastive objective for modality alignment (Ishikawa et al., 16 Jul 2025).

The architecture is situated relative to several named systems and components. The base audio-visual pretraining framework is CAV-MAE; the text branch uses the pretrained CLAP text encoder from Elizalde et al.; and the triplet-generation pipeline can use BLIP2 or LLaVa1.5 for frame-level captioning (Ishikawa et al., 16 Jul 2025). The comparative evaluations are reported against CAV-MAE and DETECLAP, while the classification baselines additionally include AudioSlowFast and Audio-MAE (Ishikawa et al., 16 Jul 2025). This suggests that LG-CAV-MAE should be understood less as a general video-LLM than as a targeted augmentation of audio-visual masked autoencoding with language-derived supervisory structure.

A central premise is that language can regularize and refine audio-visual correspondence learning. The paper’s qualitative discussion states that LG-CAV-MAE “better captures fine-grained semantic correspondences thanks to text supervision” (Ishikawa et al., 16 Jul 2025). A plausible implication is that the text modality acts as a semantic intermediary when raw audio-visual co-occurrence is ambiguous or weakly aligned.

2. Inputs, encoders, and latent representations

The model operates on three inputs. The audio input $X^a \in \mathbb{R}^{T^a \times F}$ is a log-mel spectrogram with $T^a$ time frames and $F$ mel bins. The visual input $X^v \in \mathbb{R}^{H \times W \times 3}$ is a single RGB frame resized to $H=W=224$. The text input $X^t$ is a natural-language caption describing the clip (Ishikawa et al., 16 Jul 2025).

The visual stream uses patchification into $N^v = (H/P)\cdot(W/P)$ non-overlapping $P \times P$ patches, denoted $\{x_i^v\}_{i=1}^{N^v}$. The audio stream is similarly patchified into $N^a = (T^a/S)\cdot(F/S_f)$ patches, with the example $T^a$0 in time and $T^a$1 in frequency, denoted $T^a$2 (Ishikawa et al., 16 Jul 2025). Random mask sets $T^a$3 and $T^a$4 are chosen with masking ratio $T^a$5, typically $T^a$6, and binary masks are used to form masked inputs $T^a$7 and $T^a$8 (Ishikawa et al., 16 Jul 2025).

The audio encoder is a standard Vision-Transformer-style encoder mapping $T^a$9 to an embedding $F$0. The visual encoder is a parallel ViT encoder mapping $F$1 to an embedding $F$2 (Ishikawa et al., 16 Jul 2025). A lightweight cross-modal encoder then refines each modality and jointly processes their concatenation. After this stage, mean pooling yields

$F$3

and

$F$4

For the text branch, the pretrained CLAP text encoder is adopted and its weights are frozen. It maps tokenized caption input to a sentence embedding

$F$5

with an optional small MLP projection if needed to match the $F$6-dimensional space of the audio and visual representations (Ishikawa et al., 16 Jul 2025).

These design choices establish a shared latent geometry across three modalities while retaining the masked-autoencoding backbone inherited from CAV-MAE. Because the text encoder is frozen, the learning signal concentrates on adapting the audio and visual branches, and on shaping the joint space through the contrastive objectives.

3. Reconstruction and tri-modal contrastive learning

LG-CAV-MAE preserves the masked autoencoder reconstruction pathway while augmenting it with two language-guided contrastive losses (Ishikawa et al., 16 Jul 2025). For reconstruction, the model concatenates the unmasked cross-modal tokens

$F$7

reinserts learnable $F$8 embeddings at the masked positions $F$9 to form $X^v \in \mathbb{R}^{H \times W \times 3}$0, and uses a cross-modal decoder to reconstruct both modalities:

$X^v \in \mathbb{R}^{H \times W \times 3}$1

The reconstruction loss is mean squared error evaluated only on masked patches:

$X^v \in \mathbb{R}^{H \times W \times 3}$2

This retains the masked reconstruction objective characteristic of masked autoencoding while making the decoder explicitly cross-modal (Ishikawa et al., 16 Jul 2025).

The contrastive component applies InfoNCE to three modality pairs: audio-visual, audio-text, and visual-text (Ishikawa et al., 16 Jul 2025). The losses are

$X^v \in \mathbb{R}^{H \times W \times 3}$3

$X^v \in \mathbb{R}^{H \times W \times 3}$4

and

$X^v \in \mathbb{R}^{H \times W \times 3}$5

For one positive pair $X^v \in \mathbb{R}^{H \times W \times 3}$6 with negatives from the rest of the batch, the paper gives

$X^v \in \mathbb{R}^{H \times W \times 3}$7

with temperature hyperparameter $X^v \in \mathbb{R}^{H \times W \times 3}$8 (Ishikawa et al., 16 Jul 2025).

The final objective is

$X^v \in \mathbb{R}^{H \times W \times 3}$9

with $H=W=224$0 and $H=W=224$1 in practice (Ishikawa et al., 16 Jul 2025). The weighting is notable because the language-guided losses are downweighted relative to the audio-visual contrastive term, indicating that text acts as an auxiliary supervisory signal rather than a co-equal optimization target. This suggests a deliberate attempt to preserve the inductive bias of audio-visual pretraining while injecting semantic regularization through language.

4. Automatic generation of audio-visual-text triplets

A defining feature of LG-CAV-MAE is its automatic method for assembling triplets $H=W=224$2 from unlabeled videos (Ishikawa et al., 16 Jul 2025). The procedure begins with frame-level captioning: videos are sampled at $H=W=224$3 fps to obtain frames $H=W=224$4, and each frame is passed to an image-to-text model, specifically BLIP2 or LLaVa1.5, to produce candidate captions $H=W=224$5 (Ishikawa et al., 16 Jul 2025).

The second stage scores audio-caption alignment using pretrained CLAP. The full-video audio is encoded into an embedding $H=W=224$6, and each candidate caption is encoded into $H=W=224$7 (Ishikawa et al., 16 Jul 2025). Similarity scores are computed as

$H=W=224$8

The caption-selection rule is then deterministic: choose the index $H=W=224$9 with the highest similarity score and set $X^t$0 and $X^t$1 (Ishikawa et al., 16 Jul 2025).

A further filtering step retains only the top $X^t$2 of triplets ranked by their maximum CLAP score, with $X^t$3 and the empirical finding that $X^t$4 works best (Ishikawa et al., 16 Jul 2025). The pseudocode in the paper makes the pipeline explicit: sample frames, caption them, encode the audio once, compute caption similarities, select the best-scoring frame-caption pair, sort all triplets by score, and keep the top $X^t$5 (Ishikawa et al., 16 Jul 2025).

This triplet-generation process is important because it converts unannotated video corpora into tri-modal supervision without manual labeling. The paper states that the approach yields “high-quality audio-visual-text triplets without requiring manual annotations” (Ishikawa et al., 16 Jul 2025). At the same time, the later discussion also states that triplet quality depends on the caption model and that improvements in image-to-text models should improve alignment (Ishikawa et al., 16 Jul 2025). That dependency identifies a key source of variance in the quality of the resulting supervisory signal.

5. Pretraining protocol and optimization regime

The reported pretraining data consists of VGGSound with 200 K videos and Kinetics700 with 650 K videos, from which triplets are generated using the described automatic pipeline (Ishikawa et al., 16 Jul 2025). The final pretraining set is the full VGGSound triplets plus the top $X^t$6 of Kinetics700 by CLAP score (Ishikawa et al., 16 Jul 2025). This composition reflects an explicit filtering strategy rather than uniform inclusion of all available unlabeled data.

The preprocessing pipeline is specified in detail. Visual inputs are resized to $X^t$7 with patch size $X^t$8. Audio inputs are converted to $X^t$9-bin log-mel representations with $N^v = (H/P)\cdot(W/P)$0 time-frames and patch size $N^v = (H/P)\cdot(W/P)$1. Text inputs are tokenized with CLAP’s BPE tokenizer with maximum length $N^v = (H/P)\cdot(W/P)$2 (Ishikawa et al., 16 Jul 2025). These details define the operational resolution of each modality and constrain the architecture’s patch-token sequence lengths.

The optimization hyperparameters are also fixed in the description. Training uses batch size $N^v = (H/P)\cdot(W/P)$3 video-audio-text triplets, mask ratio $N^v = (H/P)\cdot(W/P)$4 in both audio and visual branches, Adam with $N^v = (H/P)\cdot(W/P)$5, learning rate $N^v = (H/P)\cdot(W/P)$6, and no weight decay (Ishikawa et al., 16 Jul 2025). The scheduler applies linear warmup for $N^v = (H/P)\cdot(W/P)$7 K steps followed by cosine decay over $N^v = (H/P)\cdot(W/P)$8 K steps. Pretraining runs for $N^v = (H/P)\cdot(W/P)$9 epochs, approximately $P \times P$0 K updates, on $P \times P$1 GPUs (Ishikawa et al., 16 Jul 2025).

These protocol choices frame LG-CAV-MAE as a large-scale pretraining recipe rather than a narrow architecture proposal. A plausible implication is that the reported gains are tied not only to the tri-modal objective but also to data curation through CLAP-based triplet selection.

6. Empirical results, ablations, and comparisons

The evaluation covers both retrieval and classification. Retrieval benchmarks are the VGGSound test set with 2000 clips and AudioSet20K with balanced 20 000 clips, using Recall@1, Recall@5, and Recall@10 for audio-to-visual and visual-to-audio retrieval (Ishikawa et al., 16 Jul 2025). Classification benchmarks are VGGSound, reported as 309-way single-label accuracy, and AudioSet20K, reported as 517-way multi-label mean AP (Ishikawa et al., 16 Jul 2025).

The paper states that the method “significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task” (Ishikawa et al., 16 Jul 2025). The detailed comparison to prior work is summarized below.

Method	VS A→V R@1/R@5/R@10	VS V→A R@1/R@5/R@10
CAV-MAE	15.1 / 36.6 / 48.0	15.7 / 39.6 / 50.5
DETECLAP	15.2 / 39.2 / 49.5	17.4 / 43.0 / 51.7
LG-CAV-MAE (VS)	17.7 / 41.2 / 52.2	18.1 / 43.9 / 54.5
LG-CAV-MAE (+K700)	18.9 / 43.3 / 54.5	20.8 / 46.3 / 56.5

Method	VGGSound Acc. (%)	AudioSet20K mAP (%)
AudioSlowFast	52.5	–
Audio-MAE	–	37.0
CAV-MAE	58.9	38.4
DETECLAP	59.5	39.6
LG-CAV-MAE (VS)	59.2	41.1
LG-CAV-MAE (+K700)	60.1	42.8

The ablation results identify several operational sensitivities (Ishikawa et al., 16 Jul 2025). For the text-loss weight $P \times P$2, the best reported performance occurs at $P \times P$3, where VGGSound retrieval reaches $P \times P$4 and AudioSet20K reaches $P \times P$5 for R@1/R@5/R@10 (Ishikawa et al., 16 Jul 2025). Smaller or larger values degrade performance, indicating that language supervision is beneficial but requires careful weighting.

For caption-model choice, BLIP2 yields the best VGGSound audio-to-visual retrieval at $P \times P$6, while LLaVa1.5 yields slightly higher AudioSet20K R@1 at $P \times P$7 but lower R@5 and R@10 at $P \times P$8 compared with BLIP2’s $P \times P$9 (Ishikawa et al., 16 Jul 2025). The paper does not generalize beyond these reported settings, but the results support the stated observation that triplet quality depends on the caption model.

The dataset-combination and filtering study shows that Kinetics700 alone performs substantially worse than VGGSound-only pretraining, even when filtered. The strongest reported setting is VGGSound plus the top $\{x_i^v\}_{i=1}^{N^v}$0 of K700 by CLAP score, with VGGSound A→V retrieval of $\{x_i^v\}_{i=1}^{N^v}$1 and AudioSet20K A→V retrieval of $\{x_i^v\}_{i=1}^{N^v}$2 (Ishikawa et al., 16 Jul 2025). The random-versus-CLAP filtering comparison further reports that CLAP-based selection outperforms random selection at the same $\{x_i^v\}_{i=1}^{N^v}$3 retention level, on both VGGSound A→V and V→A retrieval (Ishikawa et al., 16 Jul 2025). This directly supports the claim that CLAP-based filtering is not merely a data-reduction step but a data-quality intervention.

7. Interpretation, limitations, and future directions

The qualitative examples in the paper illustrate the triplet-generation mechanism with retained samples such as a frame of “man strumming a guitar on stage,” captioned “A man is playing a guitar on stage.” with “acoustic guitar strum” audio and high CLAP score $\{x_i^v\}_{i=1}^{N^v}$4, and a frame of “dog running across yard,” captioned “A dog is running in a grassy yard.” with “dog barking/running” audio and score $\{x_i^v\}_{i=1}^{N^v}$5 (Ishikawa et al., 16 Jul 2025). These examples are used to exemplify the intended semantic alignment among frame, caption, and soundtrack.

The paper’s discussion identifies several limitations and future directions (Ishikawa et al., 16 Jul 2025). First, triplet quality depends on the caption model. Second, the current work uses only $\{x_i^v\}_{i=1}^{N^v}$6 fps frame captioning, and future work could explore multi-frame captions or video-LLMs. Third, extending to full sequence modeling rather than single frames may better capture temporal events. Fourth, more aggressive filtering through adaptive thresholds or soft weights by CLAP could yield further gains (Ishikawa et al., 16 Jul 2025).

Several potential misconceptions are clarified by the reported design. LG-CAV-MAE is not described as relying on manual text annotation; its triplets are automatically generated from unlabeled videos (Ishikawa et al., 16 Jul 2025). It also does not replace masked autoencoding with pure contrastive learning; instead, it combines masked reconstruction with three contrastive terms (Ishikawa et al., 16 Jul 2025). Nor is it presented as a fully trainable end-to-end tri-modal model in which all components are updated jointly, because the CLAP text encoder is explicitly frozen (Ishikawa et al., 16 Jul 2025).

In methodological terms, the framework indicates a specific research direction in which language supervision is used to improve audio-visual representation learning without requiring human-labeled captions. The reported results suggest that this improvement depends jointly on tri-modal objective design, automated triplet construction, and CLAP-based filtering. A plausible implication is that future gains may come as much from better data synthesis and alignment heuristics as from architectural modification alone (Ishikawa et al., 16 Jul 2025).