The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization

Abstract: Geo-localization aims to infer the geographic origin of a given signal. In computer vision, geo-localization has served as a demanding benchmark for compositional reasoning and is relevant to public safety. In contrast, progress on audio geo-localization has been constrained by the lack of high-quality audio-location pairs. To address this gap, we introduce AGL1K, the first audio geo-localization benchmark for audio LLMs (ALMs), spanning 72 countries and territories. To extract reliably localizable samples from a crowd-sourced platform, we propose the Audio Localizability metric that quantifies the informativeness of each recording, yielding 1,444 curated audio clips. Evaluations on 16 ALMs show that ALMs have emerged with audio geo-localization capability. We find that closed-source models substantially outperform open-source models, and that linguistic clues often dominate as a scaffold for prediction. We further analyze ALMs' reasoning traces, regional bias, error causes, and the interpretability of the localizability metric. Overall, AGL1K establishes a benchmark for audio geo-localization and may advance ALMs with better geospatial reasoning capability.

• The paper introduces the AGL1K benchmark that quantifies audio localizability using a weighted metric derived from state-of-the-art ALMs.
• It employs a multi-stage filtering pipeline on 1,444 geo-tagged audio clips from 72 countries to rigorously assess geographic reasoning.
• Experiments reveal that closed-source models outperform open-source alternatives, highlighting challenges in processing non-linguistic audio clues.

Benchmarking Audio-LLMs for Geo-Localization: The AGL1K Framework

Motivation and Problem Formulation

Geo-localization—mapping sensory input to geographic origins—has proven essential for tasks that require compositional reasoning and world knowledge, particularly in computer vision. While vision-based geo-localization has benefited from large-scale annotated datasets and enabled a wide array of classification and retrieval approaches, audio geo-localization suffers from a paucity of high-quality datasets with location metadata and from a lack of quantitative metrics to assess the informativeness of such audio recordings with respect to geography.

This paper introduces the AGL1K benchmark, designed to expose and methodically evaluate the geographic reasoning capabilities of state-of-the-art audio-LLMs (ALMs) using a challenging task: inferring location from sound. The authors propose the Audio Localizability metric, systematically curate data, and present comprehensive baselines and error analyses.

Figure 1: Overview of AGL1K, illustrating the audio geo-localization task, localizability, and composition.

Benchmark Construction: Data Acquisition, Filtering, and Localizability Metric

The AGL1K benchmark is constructed from 1,444 geo-tagged audio clips sourced from the crowd-sourced Aporee platform, encompassing samples from 72 countries across six continents. The pipeline for dataset construction is multi-stage, ensuring data diversity and geo-informative content:

• Initial Filtering: Four acoustic filters—RMS Energy, Spectral Flatness, Clipping Ratio, and Acoustic Complexity—are applied to exclude low-quality or uninformative recordings.
• Audio Localizability Calculation: The localizability of an audio sample is modeled as a weighted aggregation of durations spent in positive (geo-informative) and negative (geo-agnostic) sound categories. The weights are learned from the chain-of-thought outputs produced by strong ALMs, quantifying the reasoning contributions of different sound events.

  Figure 2: Overview of the benchmark construction framework. AGL1K is curated from the crowd-sourced Aporee platform, with successive filtering and informativeness quantification.

Analysis of localizability reveals strong alignment between learned feature attributions and human intuition. For instance, speech and regionally distributed sounds (e.g., rail transport, waves) are dominant positive contributors, while globally pervasive or generic sounds (rain, engine, train horn) provide limited or negative geographic clues.

Figure 3: Top Positive and Negative Categories illustrate that speech and unique regional sounds drive localizability while generic noises degrade it.

The scoring system further demonstrates interpretability, showing that humans and models are both sensitive to compositional acoustic environments: combinations of birdsong, language, and human activity yield highly localizable samples, while generic weather noise predicts poor localization.

Figure 4: Localizability Examples. Scores increase with the presence of region-specific clues such as speech or birdsong.

Model Evaluations and Quantitative Analysis

Sixteen state-of-the-art ALMs are evaluated on the AGL1K benchmark, spanning both closed-source (Gemini, GPT-4o) and open-source (Qwen3-Omni, Mimo, Phi-4-MM1) families. The evaluation metrics encompass mean distance error, hierarchical continent/country/city-level accuracy, thresholded localization accuracy, and reject rates.

Closed-source models consistently outperform open-source alternatives: Gemini 3 Pro achieves 2181 km mean error, 19% within 10 km, continent-level accuracy of 0.82, and country-level accuracy of 0.51. Mimo-audio, the best open-source model, records over 4,800 km error with substantially lower accuracy.

Linguistic content plays a central role; distance errors for speech-containing samples are halved relative to non-speech. This dependency underscores deficiencies in handling environmental and non-verbal sounds, limiting ALMs' performance on non-linguistic data.

Compositional Reasoning and Qualitative Case Analyses

The benchmark examples presented highlight the range of reasoning capabilities in modern ALMs. Leading models like Gemini 3 Pro integrate linguistic, environmental, and contextual cues robustly. For instance, in a Moroccan coastal sample, the model synthesizes evidence from Adhan, bird calls, wind, French speech, and traffic, outperforming other models that overcommit to a single clue or make generic guesses.

Figure 5: Benchmark examples showing audio clue aggregation and reasoning chains for diverse ALMs.

Regional Biases and Error Sources

Analysis of model predictions reveals persistent continental bias and confusion. Closed-source models demonstrate stronger continent-level consistency, while open-source models are prone to label collapse toward overrepresented regions, e.g., North America.

Figure 6: The continent-level prediction inequality in audio geo-localization, exposing cross-continental confusion and model bias.

Error analysis across 300 samples exposes predominant sources: language ambiguity, over-commitment to single clues, bird bias, label misidentification, and refusal. These findings suggest concrete areas for improvement, such as enhanced multilingual discrimination, evidence fusion strategies, and fairer geographic priors.

Figure 7: Error distribution across three models, categorizing failures by reasoning and perceptual bottlenecks.

Interactive Platform and Practical Implications

The authors deploy an interactive geo-localization platform based on Gradio, facilitating model evaluation, human benchmarking, and public engagement with the AGL1K dataset.

Figure 8: Screenshot of the interactive platform for audio geo-localization, enabling scalable evaluation and exploration.

Global distribution visualizations confirm diverse coverage but highlight imbalances, with European, Asian, and North American samples overrepresented compared to African, South American, and Oceanic regions.

Figure 9: The Global Distribution of AGL1K. Broad coverage across 74 countries, with uneven regional representation.

Theoretical and Practical Implications

The paper demonstrates that compositional geo-localization from audio is emergent but remains a challenging task for ALMs, particularly in the absence of clear linguistic clues. The introduction of the Audio Localizability metric advances the systematic curation and annotation of geo-informative sound data, paving the way for finer-grained multi-modal benchmarking.

Key implications include:

• Theory: This work exposes the limitations of unimodal and multimodal reasoning in current ALMs, especially in integrating non-linguistic evidence, region-specific environmental acoustics, and compositional inference. The localizability metric provides a quantifiable and interpretable proxy for the informativeness of audio samples in supervised and self-supervised learning paradigms.
• Practicality: Applications in misinformation detection, public safety, and heritage preservation are facilitated by scalable, robust audio geo-localization benchmarks. Deployment of interactive platforms and open-source evaluation code will further catalyze progress.

Additionally, model similarity and localizability variation at the continental level suggest that future work should consider regional stratification and fairness, as well as expansion toward multi-modal and few-shot reasoning scenarios.

Figure 10: Model similarity comparison across metrics, revealing high consistency among strong ALMs in contribution judgments.

Figure 11: Continent-distribution of localizability scores, indicating higher geo-informativeness in Africa, Asia, and South America.

Conclusion

The AGL1K benchmark sets a foundational standard for evaluating and advancing audio geo-localization in audio-LLMs. The systematic filtering based on the Audio Localizability metric, rigorous multi-model evaluation, and thorough error analysis disclose both strengths and shortcomings of state-of-the-art ALMs. The work advocates for targeted improvements in fine-grained perception, compositional reasoning, and regional fairness, establishing audio geo-localization as a critical benchmark for next-generation audio-language research.

Future Prospects

Advancing beyond current limitations requires integrating enhanced sound event recognition, broader cross-lingual datasets, and multi-modal fusion architectures. Expanding benchmarks to include more balanced global coverage and context augmentation can support robust world models. The presented metric for localizability could be adapted for active sample selection in large-scale training regimes, and the interactive platform offers a pathway toward human-in-the-loop model refinement and robust AI assessment. Future research may focus on open-source model parity, robust multi-modal fusion, and explainable, evidence-grounded geo-localization systems.