Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning

Abstract: Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal LLMs (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.

• The paper introduces a zero-shot, training-free SSL framework that leverages a novel Generation-Analysis-Refinement (GAR) pipeline for explicit meta-reasoning.
• The GAR pipeline integrates MLLMs to generate initial hypotheses, analyze cross-modal consistency through open-set role tagging and adaptive gating, and refine spatial localization.
• Experiments on benchmarks like VGGSound and MUSIC demonstrate significant performance gains, including improved AP and CIoU metrics compared to traditional methods.

Training-Free Sound Source Localization via MLLM Meta-Reasoning

Introduction

The paper "Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning" (2604.06824) redefines the Sound Source Localization (SSL) problem by leveraging Multimodal LLMs (MLLMs) for structured meta-reasoning, replacing the longstanding paradigm of contrastive feature matching. The work is motivated by the observation that contrastive and alignment-based SSL techniques lack explicit verification and causal reasoning, particularly in complex and open-set acoustic scenarios. By emulating human-like multi-stage reasoning, the proposed approach forms a zero-shot, training-free, and interpretable SSL framework supporting both single- and multi-source settings.

Framework Architecture: The GAR-SSL Pipeline

The proposed Generation-Analysis-Refinement (GAR) framework deploys MLLMs in a pipeline consisting of three explicit reasoning stages:

1. Generation: The model proposes initial spatial (bounding box) and semantic (audio class) hypotheses given image-audio pairs in a coarse-to-fine manner.
2. Analysis: Explicit evaluation of the plausibility and consistency between candidate sound sources and the audio using open-set role tagging, anchor voting, and statistical consensus.
3. Refinement: Adaptive gated geometric box operations fine-tune localization, integrating consensus reasoning and selectively refining only when confidence or consistency criteria are not met.

   Figure 1: Overview of the GAR-SSL meta-reasoning pipeline for sound source localization using MLLMs in three distinct stages.

All operations are implemented exclusively via prompt engineering, without additional training or supervised dataset requirements.

Figure 2: Detailed breakdown of each pipeline stage, including multi-trial analysis and gating for refinement.

Technical Advances and Methodological Details

Stage 1: Generation

The model independently predicts a spatial bounding box and a natural-language description for the likely sound-emitting object, as well as an open-vocabulary audio label with a self-reported confidence score. The decoupling of localization and audio understanding enables structured downstream consistency checks.

Stage 2: Analysis (Meta-Reasoning)

In this stage, MLLMs perform:

• Open-Set Role Tagging: Identification of all visible parts that could contribute to the sound (e.g., "bow", "violin body"), guided by both the image and audio.
• Anchor Voting: For each anchor (sound-generating component), a confidence score is assigned, enabling evidence-weighted geometric reasoning.
• Audio-Visual Consistency Scoring: Quantification of alignment between proposed bounding boxes, role-based anchors, and predicted audio class, promoting cross-modal causal interpretability.
• Adaptive Gating: A regime for skipping refinement if validations are high-confidence, optimizing inference efficiency and minimizing unnecessary adjustments.
• Multi-Trial Consensus: Robustness to stochasticity in LLM decoding is achieved via repeated trials and statistical aggregation over role tags, anchors, scores, and gating flags.

Stage 3: Refinement

Refinement occurs only when required by the gating mechanism, applying geometric transformations (delta shift, recentering, expansion, or shrinkage) derived from analysis outputs to produce a final bounding box. Every adjustment is justified by explicit semantic and anchor evidence, enhancing the traceability of the localization process.

Experimental Evaluation

Benchmarks and Metrics

• Datasets: Evaluations are conducted on VGGSound-Single, VGGSound-Duet, MUSIC-Solo, and MUSIC-Duet standard benchmarks.
• Metrics: Class-aware AP (CAP), class-aware IoU at thresholds, and AUC are reported, consistent with conventional SSL literature.
• The method uses Qwen2.5-Omni-7B as the MLLM backbone.

Quantitative Results

• Single-Source Localization: GAR-SSL achieves AP=60.5%, [email protected]=60.2%, and AUC=55.2% on VGGSound-Single, outperforming vision-based SSL and alternative MLLM baselines by significant margins. On MUSIC-Solo, AP=80.6%, [email protected]=98.5%, AUC=78.2% are reported.
• Multi-Source Localization: The method attains [email protected]=77.6% and AUC=45.8% on VGGSound-Duet, and demonstrates a 34.9% absolute improvement in [email protected] over the best prior on MUSIC-Duet.
• Ablation: Increasing the number of analysis iterations (N) yields consistent improvements across all metrics, confirming the utility of multi-trial meta-reasoning.

  Figure 3: Qualitative comparisons in the multi-source MUSIC-Duet and VGGSound-Duet test sets demonstrating superior spatial selectivity and separation.

  

  Figure 4: Visualization results on VGGSound-Single; GAR-SSL localizes true sound emitters more precisely than OA-SSL.

Main Claims and Analytical Insights

• Structured Meta-Reasoning: The GAR pipeline enforces explicit, open-set semantic verification rather than relying exclusively on feature similarity, which leads to marked gains in localization accuracy, interpretability, and error traceability.
• Training-Free Operation: All stages are achieved without any fine-tuning or additional supervision on the SSL task, establishing a competitive zero-shot baseline against fully-trained supervised alternatives.
• Efficiency and Modularity: The adaptive gating mechanism reduces computational cost by bypassing unnecessary refinement steps, and multi-trial analysis yields stable results without manual ensemble or post-processing.
• Model Dependency: Performance depends on the base MLLM used; larger MLLMs such as Qwen2.5-Omni-7B yield stronger results, indicating that advances in MLLM architecture directly benefit downstream SSL capabilities.

Implications and Future Perspectives

The proposed method demonstrates that zero-shot, training-free MLLMs equipped with explicit meta-reasoning can achieve or exceed the performance of conventional task-specific SSL systems on challenging audiovisual benchmarks. This suggests a paradigm shift for SSL from direct representation matching to iterative, explainable, and human-like cognitive inference—raising the possibility of extending to broader multimodal grounding or event understanding tasks.

Practical implications include dramatically reduced annotation and retraining requirements for SSL in robotic perception, surveillance, and audio-visual dialog. The explainable structure supports deployment in safety-critical and interactive applications where model accountability is crucial.

Potential future directions include:

• Temporal Reasoning Integration: Extending meta-reasoning across video frames for dynamic sound localization.
• Generalization to Open-World Audio-Visual Events: Handling silent or off-screen sources by further leveraging reasoning and knowledge grounding.
• Efficiency Optimizations: Reducing multi-trial inference cost, possibly with single-step or resource-adaptive reasoning strategies.
• MLLM Architecture Exploration: Systematic benchmarking across next-generation MLLM backbones and alignment methods.

Conclusion

By decomposing the SSL task into the explained GAR reasoning paradigm and leveraging MLLM cross-modal capabilities, this work yields a modular, interpretable, and training-free approach with state-of-the-art performance on established sound localization benchmarks. The findings motivate further research at the intersection of cognitive AI, cross-modal reasoning, and human-like scene understanding in AI systems.