Cross-Instance Audio-Visual Localization

• CI-AVL is a cross-modal approach that uses semantic prompts from different instances to align audio and visual data for accurate localization.
• It shifts from same-instance pairing to category-level semantic conditioning, enhancing robustness in multi-source and mismatched scenarios.
• Empirical results show improved direction-of-arrival estimation and image-region localization through explicit cross-modal alignment and selective masking.

Searching arXiv for the cited CI-AVL and related audio-visual localization papers. Cross-Instance Audio-Visual Localization (CI-AVL) denotes a family of audio-visual localization settings in which localization is conditioned not on a paired, same-instance visual observation, but on semantically related information drawn from a different instance, scene, or event occurrence. In the prompt-based formulation introduced by “VP-SelDoA: Visual-prompted Selective DoA Estimation of Target Sound via Semantic-Spatial Matching” (Chen et al., 10 Jul 2025), CI-AVL aims to estimate the direction of arrival (DoA) of a target sound source from two-channel audio using a “semantic visual prompt” drawn from a different instance within the same sound event category. In earlier alignment-centric formulations for image-level sound source localization, CI-AVL refers to localizing the visual region in an image that corresponds to audio from another instance of the same semantic category, while also rejecting silent objects, off-screen sounds, and mismatched pairs (Senocak et al., 2023). Taken together, these formulations treat CI-AVL as a shift from same-instance correspondence toward category-level semantic conditioning, explicit cross-modal alignment, and robustness to mismatched or multi-source conditions (Chen et al., 10 Jul 2025, Senocak et al., 2024).

1. Definitions and scope

The defining property of CI-AVL is cross-instance semantic correspondence. In the DoA formulation of (Chen et al., 10 Jul 2025), the system receives two-channel audio signals $x$ captured by a microphone pair and a user-specified prompt image $I$, and predicts the DoA $\theta$ of the image-aligned target sound event. The prompt image is selected from a different instance within the same sound event category as the target, so the conditioning signal specifies “what the target sound is” rather than showing the exact sound-producing instance. The model predicts a posterior over 180 discretized DoA classes:

$$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$

This formulation was introduced to address three reported limitations of existing audio-visual sound source localization methods: inability to selectively isolate the target sound source in multi-source scenarios, misalignment between semantic visual features and spatial acoustic features, and overreliance on paired audio-visual data (Chen et al., 10 Jul 2025).

In image-level sound source localization, the alignment-centric works “Sound Source Localization is All about Cross-Modal Alignment” (Senocak et al., 2023) and “Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment” (Senocak et al., 2024) define a related CI-AVL setting: given audio from one instance of a category and an image containing a different instance of that category, the model should localize the corresponding visual source region. These works emphasize that genuine localization requires cross-modal semantic understanding rather than mere objectness bias or temporal co-occurrence. They also broaden the problem by requiring abstention or down-weighting when audio-visual semantics do not match, including silent objects, off-screen sounds, and semantically mismatched events (Senocak et al., 2023, Senocak et al., 2024).

A plausible implication is that CI-AVL is not a single benchmarked task with one canonical output type, but a broader research direction spanning spatial-region localization in images and selective DoA estimation in spatial audio. The common denominator is cross-instance semantic conditioning and the need to align “what” with “where.”

2. Relationship to conventional audio-visual localization

CI-AVL departs from conventional audio-visual sound source localization chiefly in its treatment of pairing, selectivity, and semantic robustness. Conventional AV-SSL methods typically rely on paired audio-visual data in which the visual signal depicts the same instance that produces the audio. By contrast, CI-AVL in (Chen et al., 10 Jul 2025) uses category-level prompts from different instances, thereby reducing reliance on manually curated paired audio-visual data and enhancing generalization to unpaired scenarios and diverse categories beyond human speech. The same paper explicitly frames CI-AVL as identity-aware localization: semantic conditioning provides the “what,” which must be aligned with spatial cues that specify the “where” (Chen et al., 10 Jul 2025).

The alignment-centric literature argues that conventional training can induce objectness bias: models may highlight salient or “object-like” regions regardless of the audio semantics. “Sound Source Localization is All about Cross-Modal Alignment” (Senocak et al., 2023) states that purely spatial attention trained with a single positive pair per clip can fail in CI-AVL and mismatched settings, because it does not learn sufficiently strong cross-modal semantic grounding. The proposed remedy is a joint objective combining localization and instance-level alignment, with multiple positive views and conceptual positives to generalize across instances and to detect mismatches (Senocak et al., 2023).

“Localizing Visual Sounds the Easy Way” (Mo et al., 2022) operationalizes a simpler cross-instance constraint through a multiple-instance contrastive learning formulation. For each audio $a_i$, the model requires that there exist at least one spatial location in the paired image $I_i$ with high similarity while all spatial locations in non-paired images $I_j$ for $j\neq i$ remain low-similarity. The bag-level score is

$s(i\to k)=\max_p s(z_{a,i},z_{v,k,p}),$

and this supports the CI-AVL principle of “align at least one location in the paired image, not matching any location in other images” (Mo et al., 2022).

A recurring misconception in this literature is that localization quality can be judged purely by overlap metrics such as cIoU or AUC. The alignment-oriented papers argue that this is insufficient because object-guided post-processing or objectness priors can inflate overlap scores without strong semantic grounding, especially under silent, off-screen, or mismatched conditions (Senocak et al., 2023, Senocak et al., 2024). This suggests that CI-AVL is as much about mismatch rejection and semantic fidelity as it is about spatial precision.

3. Core methodological patterns

Three methodological patterns recur across CI-AVL research: semantic prompting, explicit cross-modal alignment, and selective localization under interference.

In (Chen et al., 10 Jul 2025), semantic prompting is instantiated through a prompt image $I$ encoded by CLIP into a visual semantic feature $I$0, broadcast along time to $I$1. Mixed audio $I$2 is processed by VGGish to produce semantic audio features $I$3. These are fused with a Conformer:

$I$4

The semantic audio features serve as a bridge between visual semantics and spatial audio cues, and the prompt image supplies category-level guidance about the target source (Chen et al., 10 Jul 2025).

The same paper then constructs target-selective masks in the time-frequency domain using a Frequency-Temporal ConMamba design. For the real STFT component $I$5, the frequency branch is described as

$I$6

$I$7

A Semantic-Spatial Matching module aligns the semantic multimodal feature $I$8 with the spatial feature $I$9 using an integrated Cross-Attention Conformer (CAC) and Self-Attention Conformer (SAC). CAC uses $\theta$0 as queries and $\theta$1 as keys and values:

$\theta$2

$\theta$3

SAC then refines the CAC output through self-attention:

$\theta$4

$\theta$5

The final masking stage predicts

$\theta$6

followed by

$\theta$7

and a Softmax MLP yields the DoA posterior (Chen et al., 10 Jul 2025).

In contrast, (Senocak et al., 2023) and (Senocak et al., 2024) formulate CI-AVL through dual similarities: a localization similarity $\theta$8 and an alignment similarity $\theta$9. With image feature map $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$0 and audio feature $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$1, localization similarity is the spatial average of cosine similarity:

$$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$2

Semantic alignment is computed in a projection space:

$$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$3

Training uses InfoNCE across multiple positive pairs generated by multi-view augmentation and conceptual nearest-neighbor positives, producing $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$4 audio-visual positive pairs per anchor (Senocak et al., 2023, Senocak et al., 2024).

EZ-VSL (Mo et al., 2022) adopts a more compact multiple-instance contrastive learning design. Its audio-to-visual loss is

$$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$5

with a symmetric visual-to-audio term. Max pooling over spatial locations encodes the assumption that at least one location in the correct image should match the audio (Mo et al., 2022).

Across these methods, CI-AVL methodology consistently separates category-level semantic agreement from localization geometry. This suggests that semantic alignment is treated not merely as an auxiliary benefit, but as a prerequisite for reliable cross-instance localization.

4. Datasets, benchmarks, and evaluation protocols

The prompt-conditioned DoA formulation of CI-AVL introduced VGGSound-SSL (VGG-SSL), described as comprising 13,981 spatial audio clips across 296 sound event categories, approximately 39 hours in total, and extending VGGSound with spatial audio and prompt images (Chen et al., 10 Jul 2025). Audio is two-channel spatial audio at 16 kHz, generated by convolving mono audio with simulated room impulse responses. The paper reports 10,000 RIRs and 2,000 simulated rooms. Prompts are frame images selected by CLIP-score against category text embeddings, and annotations include DoA and Object Category, with no bounding boxes or voice activity detection (Chen et al., 10 Jul 2025).

The spatial audio generation parameters reported for VGG-SSL are specific: microphone number 2, inter-microphone distance $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$6, room size $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$7, source azimuth $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$8, source distance $$\hat{p}(\theta)=\mathcal{F}(I,x\mid\Omega), \qquad \hat{\theta}=\arg\max_{\forall \theta}\hat{p}(\theta).$$9, and $a_i$0 in $a_i$1 (Chen et al., 10 Jul 2025). Compared with AV16.3, CAV3D, SSLR, AVRI, and STARSS23, VGG-SSL is described as broader in category coverage and tailored to CI-AVL with prompts and spatial audio (Chen et al., 10 Jul 2025).

The alignment-centric CI-AVL literature relies on a different benchmark ecology. Training data include VGGSound and SoundNet-Flickr-144K, while localization evaluation uses VGG-SS, SoundNet-Flickr-Test, and AVSBench S4 (Senocak et al., 2023). Extended VGG-SS and Extended SoundNet-Flickr explicitly include non-audible, non-visible, and mismatched samples to probe false positives and mismatch handling (Senocak et al., 2023). “Aligning Sight and Sound” (Senocak et al., 2024) adds IS3, a synthetic benchmark for interactive sound source localization, constructed with Stable Diffusion prompts combining two categories, human verification, and SAM-assisted CVAT segmentation masks and bounding boxes. IS3 is reported to contain 3240 images and 6480 unique audio-visual instances spanning 118 categories (Senocak et al., 2024).

The evaluation metrics differ across CI-AVL variants. For the DoA setting in (Chen et al., 10 Jul 2025), the metrics are

$a_i$2

and

$a_i$3

with $a_i$4 (Chen et al., 10 Jul 2025).

For image-region CI-AVL, metrics include cIoU, AUC, mIoU, F-score, AP, max-F1, LocAcc, and Recall@K for cross-modal retrieval (Senocak et al., 2023, Senocak et al., 2024). “Aligning Sight and Sound” (Senocak et al., 2024) additionally formalizes Adaptive cIoU and IIoU for interactive multi-source settings. This multiplicity of metrics reflects a central feature of CI-AVL: the task is evaluated not only by localization overlap, but also by semantic retrieval and rejection of invalid pairings.

5. Empirical findings

The most direct CI-AVL results for selective DoA estimation are reported in (Chen et al., 10 Jul 2025). Trained on VGG-SSL at $a_i$5 SNR and evaluated with MAE and ACC, VP-SelDoA achieves $a_i$6 and $a_i$7 at $a_i$8, outperforming audio-only and audio-visual baselines. Reported baseline results at $a_i$9 include CMAF with $I_i$0, $I_i$1; AVSELD with $I_i$2, $I_i$3; AVST with $I_i$4, $I_i$5; DGB with $I_i$6, $I_i$7; AVMLP with $I_i$8, $I_i$9; SELDNet with $I_j$0, $I_j$1; and GCC-MLP with $I_j$2, $I_j$3 (Chen et al., 10 Jul 2025). Under noise generalization, VP-SelDoA reports $I_j$4 MAE and $I_j$5 ACC at $I_j$6, and $I_j$7 MAE and $I_j$8 ACC at $I_j$9, remaining superior to the reported baselines (Chen et al., 10 Jul 2025).

Ablation results in (Chen et al., 10 Jul 2025) attribute the gains to semantic-spatial alignment and prompt conditioning. Replacing Semantic-Spatial Matching with FiLM is reported to worsen performance by $j\neq i$0 MAE and $j\neq i$1 ACC. Removing both semantic visual prompt and semantic audio features causes a severe degradation of $j\neq i$2 ACC and $j\neq i$3 MAE. Visual prompt alone or semantic audio alone helps, but using both yields the best performance, namely $j\neq i$4 and $j\neq i$5 (Chen et al., 10 Jul 2025). Replacing ConMamba with a Conformer in the masking path degrades performance by $j\neq i$6 MAE and $j\neq i$7 ACC (Chen et al., 10 Jul 2025).

For image-region CI-AVL, (Senocak et al., 2023) reports on VGG-SS that its alignment-based method achieves cIoU 39.94 and AUC 40.02 when nearest-neighbor search uses supervised encoders, and cIoU 39.20 and AUC 39.70 with self-supervised encoders. On SoundNet-Flickr-Test, it reports cIoU 79.60 and AUC 63.44 with supervised NN search, and cIoU 79.20 and AUC 63.00 with self-supervised NN search (Senocak et al., 2023). On cross-modal retrieval over VGG-SS, it reports Audio-to-Image Recall@1/5/10 of 16.47/36.99/49.00 and Image-to-Audio Recall@1/5/10 of 20.09/42.38/53.66 for the supervised NN-search configuration (Senocak et al., 2023). The paper explicitly presents stronger retrieval as evidence of better semantic alignment, which it treats as a prerequisite for CI-AVL.

“Aligning Sight and Sound” (Senocak et al., 2024) extends these results with additional metrics and synthetic interactive evaluation. Without pretrained vision and using supervised NN encoders, it reports on VGG-SS cIoU 39.94, Adaptive cIoU 54.20, AUC 40.02, and Adaptive AUC 48.18. On Flickr-SoundNet, it reports cIoU 79.60, Adaptive cIoU 86.80, AUC 63.44, and Adaptive AUC 69.02 (Senocak et al., 2024). On IS3, the same configuration yields cIoU 45.1, Adaptive cIoU 59.4, AUC 43.9, and Adaptive AUC 50.9. On Extended Flickr-SoundNet, it reports AP 64.43, max-F1 66.90, and LocAcc 79.60, and on Extended VGG-SS it reports AP 34.73, max-F1 40.70, and LocAcc 39.94 (Senocak et al., 2024). These extended-set results are particularly relevant because they quantify mismatch rejection, silent-object handling, and off-screen robustness.

EZ-VSL (Mo et al., 2022) predates the explicit CI-AVL naming in this broad sense, but its objective enforces a cross-instance negative constraint and it reports strong generalization. When trained on VGG-144k and tested on Flickr, EZ-VSL reports 83.94 CIoU and 63.60 AUC; on VGG-SS, it reports 38.85 CIoU and 39.54 AUC. In open-set evaluation on VGG-SS, it reports 37.25 CIoU and 38.97 AUC on heard classes, and 39.57 CIoU and 39.60 AUC on unheard classes (Mo et al., 2022).

6. Limitations, broader connections, and open problems

The limitations reported in (Chen et al., 10 Jul 2025) are concrete. The paper focuses on two-channel spatial audio simulated via RIRs and does not report performance on real multichannel Ambisonic recordings or broader microphone array geometries. It also defines no explicit alignment loss beyond the end-to-end objectives

$j\neq i$8

$j\neq i$9

$s(i\to k)=\max_p s(z_{a,i},z_{v,k,p}),$0

The data summary further notes plausible limitations, explicitly marked as assumptions, including possible dataset bias from RIR-simulated two-channel data, dependence on robust category-level semantics, and uncertainty under outdoor scenes, extreme reverberation or noise, or occlusions (Chen et al., 10 Jul 2025).

In the image-level CI-AVL line, the principal failure modes are related to objectness bias, quality of nearest-neighbor mining, and mismatch conditions. (Senocak et al., 2023) notes that object-guided localization can introduce false positives and degrade accurate localization, while rare or atypical categories may have poorer nearest neighbors for conceptual positive mining. It also states that subtle temporal events are handled only through small audio shifts, not full mixture modeling (Senocak et al., 2023). (Senocak et al., 2024) further reports that heavy occlusion, extreme scale differences, very noisy audio, and tightly clustered concurrent sources with similar timbre can reduce localization precision (Senocak et al., 2024).

A broader connection appears in “UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization” (Geng et al., 2024). That work uses the phrase CI-AVL in a temporal, proposal-based setting involving TAL, SED, and AVEL, where multiple overlapping instances must be localized across modalities in untrimmed video. UniAV uses ONE-PEACE visual, audio, and text encoders, an Audio-Visual Pyramid Transformer, task-specific experts, and a unified language-aware classifier derived from prompted text embeddings (Geng et al., 2024). This is not the same task as prompt-based DoA CI-AVL or image-region cross-instance sound localization, but it reinforces a common research trajectory: aligned multimodal representations, multi-instance handling, and prompt-conditioned open-vocabulary behavior.

The application domains explicitly discussed in (Chen et al., 10 Jul 2025) include human-robot interaction, AR/VR and telepresence, and selective audio-visual event localization for surveillance, smart homes, and multimedia retrieval. The claimed operational benefit is reduced dependence on tightly synchronized audio-visual capture, since cross-instance prompting uses category-level prompts rather than same-instance paired observations (Chen et al., 10 Jul 2025).

A plausible implication is that future CI-AVL work will be shaped by three unresolved issues already visible across these papers: the need for benchmarks that separate true semantic localization from objectness effects, the need for robust handling of real multi-source and multi-array audio conditions, and the need for unified formulations that reconcile region localization, DoA estimation, mismatch rejection, and open-vocabulary prompting.