DenseAV: Dense Audio-Visual Grounding

• DenseAV is a self-supervised dual-encoder model that computes detailed local cross-modal correspondences to ground sounds and spoken words in image regions.
• It employs separate backbone architectures and a multi-head aggregation operator to extract and align dense visual features and temporal audio tokens.
• DenseAV outperforms prior methods on segmentation and retrieval benchmarks by optimizing local similarity rather than relying solely on global embeddings.

DenseAV is a self-supervised dual-encoder grounding architecture for learning high-resolution, semantically meaningful, audio-visually aligned local features from paired video and audio alone. Rather than optimizing only pooled clip embeddings, it computes dense image features over spatial positions and dense audio features over time, forms a full local cross-modal similarity volume, and aggregates that volume with a multi-head operator before applying a contrastive objective. This design causes localization to emerge directly from training, enabling grounding of both sounds and spoken words to image regions without explicit localization labels and without a separate text encoder (Hamilton et al., 2024).

1. Conceptual basis and problem formulation

DenseAV addresses a central limitation of prior audio-visual contrastive learning: strong global retrieval does not guarantee that local image patches and local audio tokens are semantically aligned. The architecture is built around the claim that many systems can solve cross-modal retrieval with pooled embeddings, class tokens, or other global summaries while still failing to localize which pixels correspond to a sound or a spoken word. DenseAV therefore redefines the training signal so that local correspondences, rather than only global instance identity, determine the contrastive similarity (Hamilton et al., 2024).

The model treats speech as audio rather than text. There is no separate text encoder, and there is no explicit textual supervision during training. Spoken prompts are only used at evaluation time, where text-to-speech provides word timing so that a target utterance span can be isolated. During training, DenseAV learns from paired audiovisual data alone and is intended to ground two distinct forms of correspondence: non-speech sounds such as barks or waterfalls, and spoken words referring to visible objects.

A recurring misconception in the surrounding literature is that retrieval accuracy is a sufficient proxy for grounding quality. DenseAV directly contests this view. Its empirical program is organized around the claim that a model may retrieve correctly while its local feature maps remain too weakly aligned for semantic segmentation or sound-source localization. This suggests that the relevant distinction is not merely between good and bad embeddings, but between objectives that preserve local cross-modal structure and objectives that erase it too early.

2. Dense representations and the multi-head aggregation operator

DenseAV is a two-branch model with modality-specific backbones and lightweight aligners. The visual branch uses a DINO Vision Transformer initialized from self-supervised ImageNet pretraining, followed by a channel-wise LayerNorm and a $1 \times 1$ convolution. The audio branch uses HuBERT initialized from self-supervised speech pretraining on LibriSpeech, followed by a channel-wise LayerNorm and two $3 \times 3$ convolutions. The dense outputs are

$$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$

where $C$ is the per-head channel dimension, $K$ is the number of heads, $H,W$ are spatial dimensions, and $F,T$ are frequency and time dimensions. With HuBERT, $F=1$, so the audio representation is effectively temporal (Hamilton et al., 2024).

The core representational object is the full local similarity tensor: $$s(a,v)[k,f,t,h,w] = \sum_{c=1}^{C} a[c,k,f,t] \cdot v[c,k,h,w].$$ This tensor measures head-specific affinity between an audio event at $(f,t)$ and an image region at $3 \times 3$0. Localization maps are obtained by slicing or averaging this tensor appropriately at inference.

DenseAV’s defining operator converts this dense similarity field into a scalar contrastive score: $3 \times 3$1 The operator performs a max over spatial positions and heads, then averages over audio positions. The max over $3 \times 3$2 allows a small object to dominate even if it occupies few pixels; the average over time yields stable gradients; the max over heads allows different heads to specialize in different correspondence types. The paper explicitly presents this as a multi-head generalization of earlier max-based localization objectives.

This aggregation rule is the mechanism by which localization emerges. Because the positive-pair score is built from maxima over local similarities rather than from pooled global vectors, the model can only increase similarity by making at least some audio regions and image regions align strongly. The same construction also supports inference-time grounding: extract dense features, compute $3 \times 3$3, optionally max-pool over heads, average over the relevant audio time span, and bilinearly upsample the resulting $3 \times 3$4 heatmap to image resolution.

3. Learning objective, specialization, and stabilization

DenseAV uses a symmetric InfoNCE-style objective over image-audio pairs. For a batch $3 \times 3$5, the paper gives the audio-to-visual direction as

$3 \times 3$6

together with the symmetric $3 \times 3$7 term and a trainable inverse temperature $3 \times 3$8. The total loss is

$3 \times 3$9

The distinguishing aspect is that the scalar similarity inside InfoNCE is the dense multi-head aggregation, not a global inner product (Hamilton et al., 2024).

To encourage head specialization, the model introduces a disentanglement regularizer. For the two-head case,

$$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$0

with $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$1. This penalizes simultaneous activation of both heads at the same local positions and biases the heads toward distinct modes of audio-visual correspondence.

The supplementary implementation adds a stability term composed of four regularizers: $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$2 with

$$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$3

These terms address collapse, temperature drift, overuse of negative evidence, and temporal instability. The paper states that no single stability term dominates final performance, but together they help prevent collapse.

Training uses AudioSet and PlacesAudio. For the first 3000 steps, only the aligners are trained while DINO and HuBERT remain frozen. The full model is then trained for 800,000 additional steps. All aligner weights are trained, all HuBERT weights are fine-tuned, and DINO is adapted using LoRA on the attention $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$4, $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$5, and $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$6 layers with LoRA rank $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$7. The reported hardware setup is 8 V100 GPUs with effective batch size 80 and negatives aggregated across GPUs.

4. Benchmarks, datasets, and quantitative performance

DenseAV introduces two evaluation datasets designed specifically to test local grounding rather than only retrieval. The speech-prompted semantic segmentation benchmark is built from ADE20K masks plus synthetic spoken prompts of the form “A picture of a(n) [object]”; it contains 3030 image-object pairs spanning 478 ADE20K classes. The sound-prompted semantic segmentation benchmark is also derived from ADE20K, paired with VGGSound clips through ontology matching; it contains 106 image-object pairs across 20 ADE20K classes. Both use mAP and mIoU for binary mask prediction (Hamilton et al., 2024).

The paper reports that DenseAV improves substantially over the prior art on both benchmarks and also leads the reported retrieval comparisons. On speech-prompted segmentation, the comparison is DAVENet $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$8 mAP / $$v \in \mathbb{R}^{CKHW}, \qquad a \in \mathbb{R}^{CKFT},$$9 mIoU, CAVMAE $C$0, ImageBind $C$1, and DenseAV $C$2. On sound-prompted segmentation, the comparison is DAVENet $C$3, CAVMAE $C$4, ImageBind $C$5, and DenseAV $C$6. On PlacesAudio 1000-way retrieval, DenseAV reaches Acc@10 of $C$7 for image-to-audio and audio-to-image; on AudioSet 1000-way retrieval, the main table reports $C$8. The supplement also reports VGGSound Source localization of $C$9 cIoU / $K$0 AUC for DenseAV.

Benchmark	Prior comparison	DenseAV
Speech-prompted segmentation	DAVENet $K$1	$K$2
Sound-prompted segmentation	CAVMAE $K$3	$K$4
PlacesAudio Acc@10	CAVMAE $K$5	$K$6
AudioSet Acc@10	ImageBind $K$7	$K$8

A central ablation isolates aggregation. With the rest of the setup aligned, average pooling yields speech mAP $K$9, CLS token $H,W$0, SimPool $H,W$1, and DenseAV multi-head aggregation $H,W$2. Retrieval remains relatively high across these alternatives: average pooling $H,W$3 @10, CLS token $H,W$4, SimPool $H,W$5, and DenseAV multi-head $H,W$6. This is the paper’s clearest quantitative argument that localization quality and retrieval quality can diverge sharply.

The same section reports robustness under environmental noise added to spoken prompts. DenseAV retains $H,W$7 on the speech-grounding benchmark under noisy prompts, whereas DAVENet reports $H,W$8, CAVMAE $H,W$9, and ImageBind $F,T$0. The paper also highlights that DenseAV outperforms ImageBind on cross-modal retrieval while using less than half the trainable parameters.

5. Emergent disentanglement, interpretation, and limitations

One of DenseAV’s most distinctive findings is its unsupervised separation of “chirp” and “chat”: when trained jointly on sound-heavy AudioSet and speech-heavy PlacesAudio, a two-head model can learn one head that responds mainly to ambient or object sounds and another that responds mainly to spoken language. The heads are not manually assigned these roles. The separation emerges from max-pooling over heads, the disentanglement penalty, and the mixed training distribution (Hamilton et al., 2024).

The paper quantifies this effect using two measures. PredDis evaluates how well head activations predict whether a sample belongs to the sound dataset or the language dataset, and ActDis measures how inactive the non-dominant head is on the wrong dataset. The best DenseAV model achieves PredDis $F,T$1 and ActDis $F,T$2. An ablation further shows that head max-pooling is crucial for specialization: without $F,T$3 and without head max-pool, PredDis is $F,T$4 and ActDis is $F,T$5; without $F,T$6 but with head max-pooling, PredDis is $F,T$7 and ActDis is $F,T$8.

This result matters because it reframes DenseAV as more than a localization architecture. It is also a mechanism for discovering multiple coupling regimes inside paired audiovisual data. In the reported setting, one regime is object sound production and the other is spoken object reference. The architecture does not require explicit labels telling it which samples are “speech” and which are “sound.”

The limitations described in the paper remain substantive. Rare visual forms and rare sounds can produce diffuse maps. DenseAV may emphasize discriminative subregions rather than the full object extent. Some apparent errors reflect ontology ambiguity or segmentation-label mismatch, and compound concepts can produce ambiguous groundings. The supplement also argues that DINO is preferable to CLIP as a visual backbone because DINO’s local tokens are better behaved for localization, especially with $F,T$9 patch size.

6. Variants, comparisons, and terminological extensions

Subsequent work has extended DenseAV-style ideas into substantially different regimes. “VeS: Teaching Pixels to Listen Without Supervision” studies unsupervised speech-image alignment in a multilingual subset of Project Vaani comprising 4.83 million speech segments spanning 83 language and dialect labels. Under matched architecture and training conditions, it compares a global mean-pooled CLIP-style loss, a DenseAV-style dense token matcher, and a hybrid loss. The dense objective yields a $F=1$0 relative improvement in Audio-to-Visual R@1 over the global baseline, with A2V R@1 increasing from $F=1$1 to $F=1$2, and it produces sharp zero-shot localization heatmaps despite keeping the DINOv2-Large vision backbone entirely frozen (Raj, 29 Jul 2025).

A different line of comparison appears in “Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning.” That work is not a dense grounding method in the same sense, but it reports a VGGSound retrieval comparison against the official DenseAV model. The quoted numbers are V$F=1$3A R@1/R@5/R@10 of $F=1$4 and A$F=1$5V $F=1$6 for DenseAV, versus $F=1$7 and $F=1$8 for TG-DP. The paper’s stated focus is the reduction of semantic noise and optimization interference by decoupling reconstruction and alignment into separate masked paths, so the comparison is best read as a retrieval-oriented contrast rather than a direct replacement for DenseAV’s dense correspondence objective (Wang et al., 9 Apr 2026).

The term “DenseAV” has also broadened in the long-video localization literature, where it can denote dense audio-visual event localization rather than spatial or token-level grounding. “Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration” formulates DAVEL on UnAV-100, a benchmark of 10,790 untrimmed videos with 100 audio-visual event categories, average video length around 42.1 s, and average 2.8 audio-visual events per video. Its CCNet model reports Avg. mAP $F=1$9 against a DAVEL baseline at $$s(a,v)[k,f,t,h,w] = \sum_{c=1}^{C} a[c,k,f,t] \cdot v[c,k,h,w].$$0, and $$s(a,v)[k,f,t,h,w] = \sum_{c=1}^{C} a[c,k,f,t] \cdot v[c,k,h,w].$$1 with stronger ONE-PEACE features (Zhou et al., 2024). “CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization” then introduces a weakly supervised W-DAVEL setting and reports Avg. mAP $$s(a,v)[k,f,t,h,w] = \sum_{c=1}^{C} a[c,k,f,t] \cdot v[c,k,h,w].$$2 on UnAV-100, improving over CCNet at $$s(a,v)[k,f,t,h,w] = \sum_{c=1}^{C} a[c,k,f,t] \cdot v[c,k,h,w].$$3 under that supervision regime (Zhou et al., 6 Aug 2025).

These later developments suggest a useful distinction. In the original DenseAV architecture, density refers to dense cross-modal correspondence between image regions and audio time positions. In DAVEL and W-DAVEL, density refers to dense temporal event localization in long untrimmed videos. The shared premise is that preserving or recovering fine-grained multimodal structure is necessary when global summaries are insufficient, but the target granularity differs: spatial grounding in one case, temporal event parsing in the other.