Geo-ATBench: A Benchmark for Geospatial Audio Tagging with Geospatial Semantic Context

Abstract: Environmental sound understanding in computational auditory scene analysis (CASA) is often formulated as an audio-only recognition problem. This formulation leaves a persistent drawback in multi-label audio tagging (AT): acoustic similarity can make certain events difficult to separate from waveforms alone. In such cases, disambiguating cues often lie outside the waveform. Geospatial semantic context (GSC), derived from geographic information system data, e.g., points of interest (POI), provides location-tied environmental priors that can help reduce this ambiguity. A systematic study of this direction is enabled through the proposed geospatial audio tagging (Geo-AT) task, which conditions multi-label sound event tagging on GSC alongside audio. To benchmark Geo-AT, Geo-ATBench is introduced as a polyphonic audio benchmark with geographical annotations, containing 10.71 hours of audio across 28 event categories; each clip is paired with a GSC representation from 11 semantic context categories. GeoFusion-AT is proposed as a unified geo-audio fusion framework that evaluates feature-, representation-, and decision-level fusion on representative audio backbones, with audio- and GSC-only baselines. Results show that incorporating GSC improves AT performance, especially on acoustically confounded labels, indicating geospatial semantics provide effective priors beyond audio alone. A crowdsourced listening study with 10 participants on 579 samples shows that there is no significant difference in performance between models on Geo-ATBench labels and aggregated human labels, supporting Geo-ATBench as a human-aligned benchmark. The Geo-AT task, benchmark Geo-ATBench, and reproducible geo-audio fusion framework GeoFusion-AT provide a foundation for studying AT with geospatial semantic context within the CASA community. Dataset, code, models are on homepage (https://github.com/WuYanru2002/Geo-ATBench).

• The paper introduces Geo-ATBench, a 3,854-clip benchmark with 28 event labels and 768-dimensional OpenStreetMap-derived geospatial semantic context, plus GeoFusion-AT reference methods for early, intermediate, and late fusion.
• The benchmark shows that geospatial context improves fine-grained tagging across all tested backbones, with GeoFusion-Early-AST achieving the best result at 0.846 mAP versus 0.820 for audio-only AST, while GSC-only classification reaches 0.767 mAP.
• The paper finds that location cues help most for place-associated events such as Helicopter, which gains 0.3378 AP, but offer little or negative benefit for spatially diffuse sounds such as Speech and Laughter, highlighting the need for geographic hold-out evaluation and robust handling of uneven OSM coverage.

Motivation and task formulation

Multi-label audio tagging (AT) in computational auditory scene analysis (CASA) is conventionally formulated as an audio-only recognition problem, yet acoustic similarity between event classes—overlapping sources producing near-identical time-frequency patterns—remains a persistent source of error that waveform-only models cannot resolve. The paper argues that disambiguating cues often reside outside the waveform, specifically in location-tied environmental priors. It formalizes this observation as the geospatial audio tagging (Geo-AT) task: given a paired input of an acoustic representation $\mathbf{A}$ and a geospatial semantic context (GSC) vector $\mathbf{g}$ derived from points of interest (POI), predict a multi-label vector over a fixed event vocabulary. Crucially, the formulation is deliberately agnostic to integration mechanism; it prescribes neither architecture nor fusion point, leaving those as objects of study.

The choice of POI-derived GSC over raw GPS coordinates is motivated by interpretability and alignment: POI semantics provide structured descriptions of nearby place categories (e.g., beaches, highways, train stations) that can be encoded into dense vectors compatible with audio embeddings. The paper positions Geo-AT within the family of context-augmented sound understanding methods that use non-time-aligned auxiliary metadata, and identifies the absence of standardized tasks, benchmarks, and consistent baseline reporting in prior metadata-fusion work as the principal obstacle to reproducible progress.

The Geo-ATBench dataset

Geo-ATBench comprises 3,854 real-world polyphonic clips totaling 10.71 hours, sourced from Freesound.org and a GPS-annotated soundscape collection, restricted to recordings from 2012–2025 where geo-tagged uploads are sufficiently dense. Each data point is a triplet: a 10-second mono 16 kHz WAV clip, a multi-label vector over 28 event classes, and a 768-dimensional GSC vector constructed by querying OpenStreetMap via the Overpass API over a square region centered on the clip's coordinates using 11 OSM feature keys (land use, amenities, natural features, etc.). The resulting POI composition is serialized into descriptive strings and encoded with frozen BERT-base with mean pooling.

Annotation required approximately 800 person-hours over four months: manual listening-based labeling cross-validated against uploader tags, with disagreements re-examined and, where necessary, resolved using POI-derived cues. The 28 classes follow the AudioSet taxonomic grouping into Natural Sounds, Human Sounds, and Sounds of Things, with class counts ranging from 1,024 clips (Bird sounds) down to 42 (Truck). Intra-class log-Mel cosine similarity analysis confirms substantial acoustic confusability across classes, which is precisely the regime where GSC priors are hypothesized to help.

A methodological caveat stated explicitly by the authors: although a circular neighborhood would better match isotropic sound propagation, square bounding boxes are used for computational efficiency in OSM queries—a practical approximation applied uniformly across samples.

The GeoFusion-AT framework

GeoFusion-AT provides reference implementations at three canonical fusion points, instantiated on three AudioSet-pretrained backbones (PANNs, AST, CLAP):

• Feature-level (GeoFusion-Early): the GSC vector is linearly projected to the spectrogram's frequency resolution, broadcast across time, and channel-concatenated with the log-Mel spectrogram (for AST, injected instead as a dedicated [GSC] token). Pretrained audio-channel weights are preserved while GSC channels/tokens are zero-initialized.
• Representation-level (GeoFusion-Inter): modality-specific encoders produce clip-level embeddings combined via symmetric cross-modal attention with residual mixing, following VATT-style bidirectional refinement.
• Decision-level (GeoFusion-Late): independent audio and GSC branches produce class-wise logits fused as $$\mathbf{z}_{\text{fused}} = \mathbf{z}_{\text{audio}} + \boldsymbol{\lambda} \odot \mathbf{z}_{\text{GSC}}$$, with a learnable, softplus-constrained, zero-initialized class-specific weight vector $\boldsymbol{\lambda}$.

Training uses BCE loss, AdamW at learning rate $10^{-5}$, early stopping on validation F1, and five seeds with multi-label stratified 70/15/15 splits.

Benchmark results

GSC-only baselines are surprisingly competitive. A frozen-BERT-plus-MLP classifier operating purely on POI-derived GSC achieves mAP 0.767 on the 28-class task—essentially matching the fine-tuned audio-only PANNs baseline (0.770)—and 0.867 on the coarse-grained 3-class task. Performance increases monotonically with POI extraction range up to 1000 m, which the authors attribute partly to sparse OSM coverage at small radii and variable GPS accuracy, though they acknowledge these explanations are not exhaustively verified.

Fusion generally helps on the fine-grained task. Across all three backbones and fusion strategies, incorporating GSC improves 28-class mAP relative to audio-only baselines, with Welch t-tests confirming significance for AST+early ($p<0.05$), PANNs+late ($p<0.001$), and CLAP+intermediate ($p<0.01$):

Strategy	PANNs	AST	CLAP
Audio-only	0.770±0.006	0.820±0.015	0.824±0.008
GSC-only	0.767±0.010	—	—
Early fusion	0.812 (+0.042)	0.846 (+0.026)	0.826 (+0.002)
Intermediate fusion	0.824 (+0.054)	0.829 (+0.009)	0.842 (+0.018)
Late fusion	0.833 (+0.063)	0.843 (+0.023)	0.831 (+0.007)

The best fine-grained result is GeoFusion-Early-AST (mAP 0.846), statistically indistinguishable from GeoFusion-Inter-CLAP ($p>0.5$). Notably, the backbone ranking reverses between zero-shot and fine-tuned regimes: PANNs dominates zero-shot inference under Word2Vec-based AudioSet-to-Geo-ATBench label mapping (micro AUC 0.8576 vs. 0.6672 for AST), but AST leads after fine-tuning. The authors attribute this to label-mapping calibration effects and domain shift rather than intrinsic representational differences—an interpretation that underscores how fragile cross-dataset zero-shot comparisons can be.

Per-label analysis localizes the benefit. Using GeoFusion-Early-AST as an exemplar, 17 of 28 classes gain more than 5% AP from GSC ("GSC-benefiting"), led by Helicopter with $\Delta AP = 0.3378$ (~+52.62% relative)—consistent with helicopters being strongly tied to specific place semantics. Nine classes are GSC-neutral (within ±5%), including Bell, Singing, and Footsteps. Two classes, Speech and Laughter, degrade below −5%, plausibly because these vocalizations are spatially ubiquitous. Explosion shows near-zero change, which the authors connect to their Freesound sample being dominated by fireworks rather than industrial events—a dataset-composition effect worth noting when generalizing per-label conclusions.

On the supplementary 3-class coarse-grained task, gains are smaller and inconsistent (late fusion hurts PANNs by −0.012), suggesting GSC contributes most when fine-grained acoustic ambiguity is present rather than at coarse semantic granularity.

Human evaluation

A within-subject listening study with 10 participants annotating 579 clips against the 28-class checklist yields high raw agreement (mean 0.97) but only moderate chance-corrected reliability (Krippendorff's $\mathbf{g}$0 over 16,212 items), reflecting both class sparsity (~4.5% positive rate) and genuine perceptual variability on polyphonic material. Majority-vote consensus (≥5/10 "exist") serves as the human reference. Paired Wilcoxon signed-rank tests over 28 per-class F1 scores show no significant difference between model performance evaluated against Geo-ATBench labels versus human consensus for either audio-only CLAP (F1 0.628 vs. 0.570; $\mathbf{g}$1, $\mathbf{g}$2) or GeoFusion-Inter-CLAP (0.649 vs. 0.592; $\mathbf{g}$3, $\mathbf{g}$4). This supports the benchmark labels as human-aligned, though the moderate inter-rater reliability warrants cautious interpretation for low-prevalence events, as the authors themselves note.

Limitations and open questions

Several constraints bound the reported conclusions. First, evaluation uses clip-level random splits with location-derived semantic context, not strict geographic hold-out; since similar POI compositions recur across locations, reported results measure generalization under semantic-context overlap rather than spatial extrapolation to unseen regions. Second, the GSC representation depends on OSM completeness, which is known to be spatially uneven, and on consumer-grade GPS accuracy; both factors confound the range-sensitivity analysis. Third, the square-versus-circular query geometry is a pragmatic approximation whose effect on GSC quality is not quantified. Fourth, the human study involves ten participants sharing a similar linguistic and cultural background, limiting claims about annotation generality. Finally, the fusion comparison covers three backbones and three fusion points but does not exhaust the design space—for instance, frame-level or token-level cross-modal conditioning beyond the [GSC] token remains unexplored, and no significance testing is reported for the coarse-grained results, where several deltas fall within one standard deviation.

Conclusion

This paper contributes a coherent triad: a standardized Geo-AT task definition conditioning multi-label tagging on POI-derived GSC, an open 3,854-clip benchmark with ~800 person-hours of curated annotation, and a reproducible fusion framework spanning feature-, representation-, and decision-level integration over three pretrained backbones. The central empirical finding—that GSC improves fine-grained tagging across all tested configurations, with large gains concentrated on location-tied events such as Helicopter and no benefit for spatially diffuse events such as Speech—establishes when geospatial priors complement acoustic evidence. The human alignment study strengthens the benchmark's validity. Open questions include geographic hold-out evaluation, robustness to OSM coverage gaps, and whether finer-grained fusion mechanisms can further exploit location-tied semantics.