SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios

Abstract: Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning-based systems. Sound source localization (SSL) has achieved remarkable success with deep learning, yet most methods localize all active sources without selectivity. Conversely, target sound extraction (TSE) extracts sources using multimodal prompts but typically fails to preserve the multichannel spatial information required for accurate localization. To bridge this gap, we formulate the task of prompt-guided selective target sound localization and propose SelectTSL, an end-to-end architecture that localizes only the user-specified target in multi-source acoustic scenes. Specifically, we design a target-aware selective localization strategy that employs a Prompt-Guided Selective Attention Module (PGSA) to generate prompt-informed embeddings. These embeddings guide an inter-channel phase difference (IPD) enhancer to refine raw phase cues, fusing with target magnitudes to jointly estimate direction of arrival (DoA) and target-source cardinality, i.e., the number of target sound sources. This coupled design effectively focuses on the user-specified target spatial cues for selective localization and also handles time-varying numbers of target sources. Extensive experiments on both synthetic data and real-world recordings demonstrate that our proposed method consistently outperforms other baselines and exhibits robust generalization to real acoustic environments.

• The paper introduces a novel architecture leveraging prompt-guided selective attention to generate extraction-informed embeddings for target sound localization.
• The paper employs an IPD enhancer and progressive temporal modeling to refine spatial cues and achieve superior localization accuracy, with MAE of 0.98° in small rooms.
• The paper validates its approach on synthetic and real datasets, outperforming multiple baselines in F1, MOTA, and trajectory estimation under dynamic and noisy conditions.

Prompt-Guided Selective Target Sound Localization with SelectTSL

Overview and Motivation

Sound source localization (SSL) is essential in multi-microphone audio analysis, enabling applications such as beamforming, spatial speech enhancement, and far-field ASR. While state-of-the-art SSL algorithms estimate the direction-of-arrival (DoA) for all active sources, their semantic agnosticism produces non-selective DoA trajectories that lack identity-aware localization capability. Human perception, in contrast, readily enables selective spatial attention, localizing sound sources of interest amidst clutter and competing signals.

Prompt-guided approaches for target sound extraction (TSE) have achieved controllable separation via text or audio queries, but do not preserve the spatial cues necessary for precise localization. SelectTSL addresses this critical gap by introducing an end-to-end architecture for selective target sound localization, conditioned on user prompts and robust to dynamic, noisy, and multi-source acoustic environments (2607.02343).

Figure 1: Conventional SSL localizes all sources, producing non-selective DoA trajectories (left); SelectTSL uses prompts to focus on a user-specified target, yielding selective DoA traces (right).

SelectTSL Architecture and Prompt-Guided Selectivity

At the core of SelectTSL is a Prompt-Guided Selective Attention (PGSA) module, which fuses text and/or audio cues with the input mixture to generate extraction-informed embeddings (EIEs) that emphasize the target source. These embeddings serve as semantic filters, purifying mixture representations for subsequent spatial analysis while retaining inter-channel cues.

Figure 2: SelectTSL architecture integrates a PGSA module for target-aware embeddings, an IPD enhancer for robust spatial cue refinement, and DoA and cardinality estimation heads.

The key architectural components are:

• PGSA: Conditions mixture spectrograms on CLAP-based audio/text prompt embeddings, injecting semantic selectivity via feature-wise linear modulation (FiLM) and cross-attention. Dual-path RNN blocks capture long-range context; extraction-informed embeddings (EIEs) provide instance-level guidance.
• IPD Enhancer: Refines the inter-channel phase difference (IPD) cues by gating acoustic features with semantic EIE context, denoising and reweighting the spatial evidence toward target-consistent regions.
• Progressive Refinement Temporal Modeling: Combines enhanced IPD, ILD, and extracted target magnitudes, fusing spatial and spectral cues before temporal modeling with TCNs and BiGRUs.
• Prediction Heads: Decouple frame-wise DoA posteriorgram estimation from cardinality (number of active target sources), enabling multi-target tracking and robust adaptation to time-varying target activity.

This pipeline enforces strong semantic-spatial binding, robustly tracking user-specified sources even under heavy interference, reverberation, and anisotropic motion.

Experimental Benchmark and Results

SelectTSL is thoroughly evaluated on a large-scale synthetic and real-room dataset with dual-microphone mixtures, dynamic target and interferer motion, and a broad set of target classes. Extensive baselines span track-wise (IPDNet, EINV2), non-track-wise SELD/DoA (SALSA-Lite, FN-SSL, SRP-DNN), and prompt-based (e.g., SEL [zhao2024text]) methods, all adapted to stereo configuration.

Strong numerical results are reported:

• Frame-level DoA error (MAE): 0.98°
• Frame-level F1: 95.7%
• Trajectory-level MOTA*: 91.6%
• OSPA-T: 2.1°
• SelectTSL exceeds all baselines, including prompt-based SEL, by large margins in both static and dynamic metrics—even when baselines receive oracle-clean or PGSA-filtered input.

Crucially, SelectTSL supports multi-target per prompt, handling up to two concurrent targets per query with accurate cardinality and trajectory tracking. Systematic ablation confirms the indispensability of prompt-guided coupling (vs. mixture input), IPD enhancement, and combined semantic-acoustic spatial fusion.

Visualization and Analysis

Heatmap and polar trajectory visualizations elucidate SelectTSL's behavior under various prompt regimes.

Figure 3: DoA posterior heatmaps: full (text+cue) prompt yields the most precise and continuous localization, while text- or audio-only show partial degradation and no-prompt fails under ambiguity.

SelectTSL leverages complementary strengths of text and audio cues; text-only prompts degrade gracefully with lower similarity, while audio-only conditioning is sensitive to temporal misalignment but robust to moderate time-stretch or intra-class variation.

Figure 4: Polar plots display robust DoA trajectory estimation over time, preserving continuity through source motion, silence, and intermittent overlap.

Robustness, Generalization, and Theoretical Implications

SelectTSL demonstrates substantial generalization:

• Robust to reverberation (MAE increases from 0.98° in small rooms to ~3.0° in large/reverberant spaces)
• Resilient to rapid target motion, non-stationary backgrounds, and prompt sparsity
• Maintains MAE ≈ 2.6° on real room (TAU-SRIR) data, with performance trends correlated to room geometry and acoustic conditions

The explicit decoupling of semantic selection from spatial localization, combined with robust multi-cue fusion, offers a strong architectural paradigm for user-intent-driven spatial audio processing. The work demonstrates that spatial localization can move beyond blind geometric inference towards fully controllable, context-aware, and temporally consistent scene parsing.

Implications and Future Directions

SelectTSL's prompt-guided selective localization fundamentally augments both theoretical and applied SSL/TSE paradigms. On the theoretical side, it offers a formal solution to the long-standing "what-where" gap: enabling selective spatial attention via multimodal prompts that bind semantic identity to spatial tracks. Practically, SelectTSL enables user-driven spatial filtering, dynamic AVASR, and fine-grained control of spatial scene understanding—critical for robotics, hearing augmentation, surveillance, and immersive audio.

Anticipated future developments include:

• Integration of additional modalities (visual, gesture, scene context) into multimodal prompt fusion
• Real-time deployment on embedded and edge platforms
• Extension to variable and unknown target cardinality without upper bounds
• Domain adaptation with real in-the-wild background and diverse microphone geometries

Conclusion

SelectTSL provides a technically comprehensive and empirically validated framework for prompt-guided selective sound localization, combining semantic awareness, robust spatial processing, and dynamic tracking in complex acoustic scenes. It establishes new performance benchmarks in both synthetic and real-world settings, and its architectural principles pave the way for next-generation, user-controllable auditory scene analysis in AI systems.

(2607.02343)