VP-SelDoA: Audio-Visual Selective DoA

• VP-SelDoA is an audio-visual sound source localization method that uses a cross-instance visual prompt to condition selective DoA estimation in multi-source mixtures.
• It fuses audio and visual semantic features through modules like Frequency-Temporal ConMamba and Semantic-Spatial Matching to isolate target sounds before localization.
• Empirical results show improved MAE and ACC under noisy conditions, establishing a new benchmark with the large-scale VGGSound-SSL dataset.

Searching arXiv for the specified paper to ground the article and citation. arxiv_search.query({"search_query":"id:(Chen et al., 10 Jul 2025)","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) arxiv_search.query({"search_query":"VP-SelDoA Visual-prompted Selective DoA Estimation of Target Sound via Semantic-Spatial Matching","max_results":10,"sort_by":"relevance","sort_order":"descending"}) VP-SelDoA is an audio-visual sound source localization method introduced for Cross-Instance Audio-Visual Localization (CI-AVL), a setting in which a model receives a two-channel audio waveform $\mathbf{x}\in\mathbb R^{2\times T}$ and a user-specified RGB image $\mathbf{I}$ showing a different instance of the same sound-event category, and predicts the direction-of-arrival $\theta\in[0^\circ,180^\circ]$ of the target sound event. The method is designed for multi-source mixtures in which localization must be selective rather than category-agnostic: the model must identify the sound whose semantics match the visual prompt. In the formulation reported for VP-SelDoA, DoA is discretized into $C=180$ integer bins, the target posterior is represented as a Gaussian-shaped vector, and the network $\mathcal F$ maps $(\mathbf{x},\mathbf{I})$ to a posterior $\hat p(\theta)$ whose maximum determines the estimated DoA (Chen et al., 10 Jul 2025).

1. Problem formulation and CI-AVL setting

The work situates VP-SelDoA within audio-visual sound source localization (AV-SSL), but departs from conventional paired audio-visual formulations by introducing CI-AVL. The stated motivation is that existing AV-SSL methods encounter three major challenges: inability to selectively isolate the target sound source in multi-source scenarios, misalignment between semantic visual features and spatial acoustic features, and overreliance on paired audio-visual data (Chen et al., 10 Jul 2025).

CI-AVL is defined as follows. A prompt image is not required to be frame-level paired with the target audio. Instead, it is an image from a different instance of the same sound-event category. The task is therefore to localize, within a mixture, the sound event that semantically matches the prompt image. This formulation suggests a shift from instance correspondence to category-level semantic guidance. A plausible implication is that the approach aims to retain semantic conditioning while relaxing the requirement for synchronized visual observations.

The target posterior is given by

$$p(\theta)=\exp\!\Bigl(-\tfrac{|\hat\theta-\theta|}{\sigma_\theta}\Bigr), \quad \hat\theta\in\{1,\dots,180\},$$

where $\sigma_\theta$ controls angular smoothness. The model prediction is

$$\hat p(\theta)=\mathcal F(\mathbf{x},\mathbf{I}\mid\Omega), \quad \hat\theta=\arg\max_\theta \hat p(\theta).$$

This setup makes the learning target a posterior over discrete angular bins rather than a single scalar regression value. Within the paper’s terminology, VP-SelDoA is specifically a method for selective DoA estimation of the one sound in a mixture that matches the visual prompt.

2. Architectural organization

VP-SelDoA consists of three stages: Semantic-Level Modality Fusion, Target-Selective Masking, and DoA Prediction (Chen et al., 10 Jul 2025). The architecture is organized so that semantic information is injected before spatial filtering, and the resulting spatial representation is then used to infer the DoA posterior.

In the Semantic-Level Modality Fusion stage, the audio semantic feature is obtained from a pretrained VGGish encoder,

$\mathbf{I}$0

while the visual semantic feature is obtained from CLIP,

$\mathbf{I}$1

The visual feature is expanded to $\mathbf{I}$2 and concatenated with the audio feature. A Conformer then produces the fused prompt feature

$\mathbf{I}$3

This stage is explicitly described as semantic-level fusion. Its role is not spatial localization by itself, but the construction of a prompt-conditioned latent representation that can later guide target-selective masking. Because the visual prompt comes from a different instance, this fusion is category-conditioned rather than frame-aligned. The method therefore treats semantic alignment and spatial localization as related but distinct subproblems.

3. Frequency-Temporal ConMamba and Semantic-Spatial Matching

The Target-Selective Masking stage operates on the real part of the short-time Fourier transform,

$\mathbf{I}$4

Its first submodule is the Frequency ConMamba Layer. The microphone axis is collapsed by a linear layer to $\mathbf{I}$5, after which two parallel projections are computed: $\mathbf{I}$6 A local 1D-conv + SiLU is followed by a State-Space Model (SSM) and LayerNorm,

$\mathbf{I}$7

and element-wise modulation yields the frequency-wise feature

$\mathbf{I}$8

The paper then introduces Semantic-Spatial Matching to refine $\mathbf{I}$9 under guidance of $\theta\in[0^\circ,180^\circ]$0 (Chen et al., 10 Jul 2025). This mechanism contains two Conformer blocks. In the Cross-Attention Conformer (CAC), queries are derived from the fused prompt feature $\theta\in[0^\circ,180^\circ]$1, while keys and values are derived from the frequency-wise feature $\theta\in[0^\circ,180^\circ]$2. The output is passed through Conv+FFN. In the Self-Attention Conformer (SAC), the aligned representation produced by CAC is projected again into $\theta\in[0^\circ,180^\circ]$3 and processed through self-attention followed by Conv+FFN.

The function of this module is described as bridging heterogeneous semantic and spatial features through integrated cross- and self-attention mechanisms. In the paper’s terms, it addresses the “what” versus “where” discrepancy: the prompt feature encodes semantic identity, whereas the acoustic representation encodes spatial structure. The ablation results reported later indicate that this matching mechanism is materially different from simple concatenation or FiLM conditioning.

After semantic-spatial alignment, a Temporal ConMamba Layer reshapes $\theta\in[0^\circ,180^\circ]$4 to $\theta\in[0^\circ,180^\circ]$5 and applies another ConMamba. A small MLP produces the time-frequency mask

$\theta\in[0^\circ,180^\circ]$6

The mask is applied to the real spectrogram,

$\theta\in[0^\circ,180^\circ]$7

where the masked real part is concatenated with the original imaginary part $\theta\in[0^\circ,180^\circ]$8. This design makes the masking stage explicitly target-selective: it isolates the sound implied by the prompt before final DoA inference.

4. DoA prediction and optimization objective

The final DoA estimator is a simple prediction head operating on $\theta\in[0^\circ,180^\circ]$9. A final MLP+Softmax predicts

$C=180$0

The network is therefore trained to map a prompt-conditioned, masked acoustic representation into a posterior over the 180 discretized DoA bins (Chen et al., 10 Jul 2025).

Training uses two losses. The reconstruction term is an MSE between the masked spectrogram and the ground-truth spectrogram,

$C=180$1

The DoA posterior term is an MSE between the predicted posterior and the Gaussian-shaped target posterior,

$C=180$2

The joint objective is

$C=180$3

This optimization strategy couples source isolation and localization. The reconstruction term supervises the target-selective masking pathway, while the DoA posterior term supervises angular prediction. A plausible implication is that the method treats localization accuracy as dependent on effective target purification, rather than relying exclusively on direct angle classification.

5. Dataset construction and benchmark context

To facilitate CI-AVL research, the work constructs a large-scale dataset referred to in the abstract as VGG-SSL and in the detailed description and tables as VGGSound-SSL (Chen et al., 10 Jul 2025). The reported dataset contains 13 981 spatial audio clips, about 39 h of audio, and 296 sound-event categories. The audio is 2-channel spatial audio generated via convolution with 10 000 random RIRs. Prompt images are selected per video by highest CLIP cosine-similarity to the event’s text label. The annotations are DoA, discretized to $C=180$4–$C=180$5, plus object category.

The paper compares this benchmark with prior AV-SSL datasets. The comparison emphasizes scale, number of events, and annotation types.

Dataset	Samples / Events	Annotations
AV16.3	43 / 2	DoA, VAD
SSLR	6 622 / 2†	DoA, VAD
STARSS23	168 / 13	DoA, VAD, Category
VGGSound-SSL	13 981 / 296	DoA, Category

The notation “2†” for SSLR is explained as synthetic grouping of classes. Relative to the listed baselines, VGGSound-SSL is the largest dataset in the table by number of samples and categories. This suggests that the benchmark is intended not only for method evaluation but also for studying generalization under broader category coverage.

The reported training protocol mixes two clips from different categories at 0 dB SNR and randomly selects a same-class prompt image. Optimization uses AdamW with learning rate $C=180$6 for 140k steps, batch size $C=180$7, and STFT with 64 ms window and 32 ms hop. Evaluation uses two metrics: $C=180$8 for which lower is better, and ACC@5°, the fraction satisfying $C=180$9, for which higher is better.

6. Empirical results, ablations, and limitations

The main quantitative results are reported at $\mathcal F$0 dB, $\mathcal F$1 dB, and $\mathcal F$2 dB SNR (Chen et al., 10 Jul 2025). At $\mathcal F$3 dB, VP-SelDoA achieves MAE $\mathcal F$4 and ACC $\mathcal F$5. At $\mathcal F$6 dB, it achieves MAE $\mathcal F$7 and ACC $\mathcal F$8. At $\mathcal F$9 dB, it achieves MAE $(\mathbf{x},\mathbf{I})$0 and ACC $(\mathbf{x},\mathbf{I})$1. These values are higher-performing than the listed audio-only and AV baselines across all three SNR settings.

SNR	MAE ↓	ACC ↑
0 dB	12.04	78.23
-5 dB	17.29	67.31
-10 dB	23.32	59.39

The comparison set includes SELDNet, GCC-MLP, MLP-AVC, AVMLP, DGB, AVST, AVSELD, CMAF, and VP-Conformer. The strongest directly related internal baseline is VP-Conformer, which achieves MAE $(\mathbf{x},\mathbf{I})$2 and ACC $(\mathbf{x},\mathbf{I})$3 at $(\mathbf{x},\mathbf{I})$4 dB. The gap between VP-Conformer and VP-SelDoA is consistent with the paper’s claim that the Frequency-Temporal ConMamba architecture and Semantic-Spatial Matching improve selective localization.

The ablation study on VGGSound-SSL at $(\mathbf{x},\mathbf{I})$5 dB further isolates two design choices. First, Semantic-Spatial Matching improves over simple concat and FiLM. The paper summarizes this as “Semantic-Spatial Matching vs. simple concat/FiLM (+11.5 % ACC).” Second, the prompt configuration matters substantially. Under SS-Matching, the reported results are: “–/–” with MAE $(\mathbf{x},\mathbf{I})$6, ACC $(\mathbf{x},\mathbf{I})$7; “–/A” with MAE $(\mathbf{x},\mathbf{I})$8, ACC $(\mathbf{x},\mathbf{I})$9; “V/–” with MAE $\hat p(\theta)$0, ACC $\hat p(\theta)$1; and “V/A” with MAE $\hat p(\theta)$2, ACC $\hat p(\theta)$3. The paper characterizes this as showing the importance of visual prompt and audio prompt bridge.

The stated contributions are fourfold: definition of CI-AVL as a task for selective DoA estimation guided by cross-instance visual prompts; proposal of VP-SelDoA with semantic-level fusion, Frequency-Temporal ConMamba, Semantic-Spatial Matching, and time-frequency masking; release of VGGSound-SSL with 13 981 clips and 296 classes; and empirical superiority over state-of-the-art AV and audio-only methods in MAE and ACC, including robustness under low SNRs (Chen et al., 10 Jul 2025).

The reported limitations are equally specific. Potential limitations include reliance on the quality of pretrained semantic encoders, namely VGGish and CLIP, and the use of simulated RIRs. The paper therefore identifies two future directions: testing in real-recorded multi-source environments and exploring fully end-to-end joint training of semantic and spatial backbones. These caveats qualify a possible misconception that cross-instance prompting alone removes all dependence on dataset construction or pretrained representations. The method reduces the need for frame-level audio-visual pairing, but it still depends on pretrained semantic encoders and simulation choices in the benchmark pipeline.