Probing Spatial Structure in Pretrained Audio Representations

Abstract: Pretrained spatial audio encoders are increasingly used as general-purpose representations for perceptual tasks, yet their spatial encoding capabilities remain poorly understood. We introduce the Spatial Audio Representation Learning (SARL) benchmark, a controlled framework for evaluating spatial information in pretrained audio models. SARL probes source-level factors (azimuth, elevation, distance, class) and room-level factors (RT60, volume, shape). Experiments across diverse encoders reveal three patterns: input configuration and training paradigm shape spatial encoding; source factors are consistently easier to decode than room factors; and sensitivity analysis under controlled perturbations shows heterogeneous responses to source and room variation. These results reveal systematic biases in current pretrained audio representations. SARL is released as an open-source benchmark for reproducible evaluation of spatial audio representations.

• The paper introduces the SARL benchmark to isolate spatial encoding using shallow linear probes on frozen audio representations.
• It demonstrates that multi-channel models, especially FOA encoders, outperform mono systems while revealing a systematic source–room decodability gap.
• It highlights how supervised models excel at localizing sources but struggle with global room properties, guiding future improvements in spatial audio modeling.

Probing Spatial Structure in Pretrained Audio Representations

Introduction and Motivation

The paper "Probing Spatial Structure in Pretrained Audio Representations" (2606.05544) confronts a core issue in spatial audio representation modeling: the opaque nature of spatial factor encoding in modern, pretrained audio models. As spatial audio becomes increasingly pivotal across immersive media, embodied AI, and acoustic scene analytics, the ability to analyze and diagnose what spatial information is encoded — and how — in foundation models becomes critical.

Despite the widespread adoption of spatial audio encoders, prevailing evaluation practices are highly pipeline-dependent, bound to specific downstream tasks and architectures. This makes it difficult to decouple the representational capacity of these encoders from confounding architectural or supervisory factors. Traditional benchmarking protocols (e.g., DCASE, LOCATA, MARBLE, HEAR) are largely task-centric and either focus on semantic tasks in non-spatial settings or conflate spatial representation quality with downstream design.

To address these limitations, the paper introduces the Spatial Audio Representation Learning (SARL) benchmark: a synthetic, simulator-based evaluation framework employing probing-based diagnostics. Probing, in this context, isolates the linear decodability of spatial factors from frozen, pretrained representations via shallow classifiers, thus providing a robust, architecture-agnostic way to analyze representational content.

SARL Benchmark: Controlled Spatial Probing

SARL is constructed around a controlled, simulation-based dataset with fine-grained, independent manipulation of spatial scene factors. It incorporates both source-level (azimuth, elevation, distance, class) and room-level (RT60, volume, shape) factors. Each factor can be independently perturbed, and scenes are rendered in mono, stereo, binaural, and first-order ambisonics (FOA), ensuring detailed input format analysis.

Scene data is derived from a balanced set of seven source event classes, with room and source conditions sampled to maximize control and disjointness across splits. For each probing task, frozen embeddings are extracted from the encoder backbone, pooled to the utterance level, and evaluated via a linear probe trained for classification (categorical factors) or regression (continuous factors, discretized into bins with Gaussian soft labels).

The probing protocol ensures all models — regardless of training paradigm or input format — are assessed under matched acoustic and evaluation conditions. Performance is normalized for improvement relative to a random predictor, allowing cross-task and cross-model comparison.

Figure 1: Probing performance across spatial factors shown as improvement over a random predictor baseline for various models categorized by input format and training paradigm.

Model Selection and Evaluation Targets

The benchmark encompasses a spectrum of state-of-the-art, pretrained audio encoders divided along two axes:

• Input Format: mono, stereo, binaural, FOA.
• Training Paradigm: supervised (SELD, S-AST, EINv2), self-supervised learning (SSL; e.g., GRAM, AudioMAE, SFD, W-JEPA, AVSA), and codec-based (EnCodec, BANC, SR-VAE).

This model diversity supports granular analysis of how architectural choices and learning objectives influence spatial representation content.

Numerical Findings and Key Trends

Input Format and Spatial Encoding

A principal finding is that input format substantially impacts spatial encoding capacity. Multi-channel representations (binaural, FOA) consistently outperform mono and stereo formats across tasks. Notably, FOA encoders provide an additional advantage for elevation and global room properties, attributed to their spherical harmonic basis.

Mono models such as A-MAE remain competitive on several room-level and source classification tasks, highlighting that some aspects of environmental structure (e.g., reverberation) are inferable from monaural temporal-spectral statistics even without explicit spatial channels.

Training Paradigm and Task-Specific Information

Supervised localization models display strong performance for directional source localization (especially azimuth), consistent with their objective-driven bias towards spatial cues. However, these models demonstrate markedly poor performance on room-level properties (RT60, volume, shape), indicating a representational bottleneck: they saturate on spatial location but neglect global acoustic context.

In contrast, self-supervised objectives — especially those reconstructing masked spectrogram patches (e.g., GRAM variants) — yield more balanced spatial representations, preserving both source and room factors in a decodable form. Embedding-level prediction and feature distillation (W-JEPA, SFD) in SSL, however, are less consistently effective. Codec-based models exhibit weak overall probing performance, evidencing a failure to encode linearly decodable spatial structure under perceptual fidelity/bandwidth objectives.

Source–Room Decodability Gap

A significant outcome is the systematic discrepancy between source-level and room-level factor encoding. In all evaluated encoders, source-related information (localization, class) is substantially more accessible to linear probes than room-related information, regardless of architecture, supervision, or input format. This result is robust and recurring.

Figure 2: Aggregated probing performance across factor groups after baseline normalization, illustrating the systematic gap favoring source over room factor decodability.

A plausible explanation addressed by the authors is that the training and input data for these encoders natively privilege inference of source factors (which are locally observable) over global room properties (which in general require multi-view or multi-source evidence). Pretraining protocols, typically single-source, may inherently bias models towards representations aligned with local, rather than global, spatial attributes.

Representation Sensitivity and Structural Analysis

While probe decodability reflects what can be "read out" linearly from representations, the geometric structure of embedding spaces under controlled perturbation remains underexplored. The paper introduces a representation sensitivity analysis, computing the normalized change in embedding under perturbations of source factors (position or class) versus room factors (RT60, shape, or volume), normalized by typical inter-embedding distances.

Across nearly all models, source perturbations yield larger embedding changes than room perturbations, in line with the probe results. Importantly, there is no monotonic relationship between representation sensitivity and probe decodability; some models (e.g., GRAM-F, EINv2) achieve high decodability with relatively low sensitivity, suggesting efficient, structured encoding of spatial factors. Others display large embedding shifts inconsistent with their linearly accessible information content.

Figure 3: Representation sensitivity to controlled source versus room perturbations; source sensitivity nearly always exceeds room sensitivity, especially in high-performing models.

Implications, Limitations, and Future Directions

These findings delineate systematic representational biases in contemporary pretrained spatial audio models, with profound implications:

• Practical: Diagnostic probing can inform the selection of pretrained encoders for deployment in varied spatial audio applications, including scenarios requiring nuanced awareness of room context.
• Theoretical: The pronounced source–room gap exposes representational inductive biases — potentially resulting from pretraining data, architectural priors, or learning objectives — and highlights the inadequacy of current paradigms for global acoustic environment modeling.
• Methodological: The SARL framework, with its synthetic, factor-controlled dataset and probing protocol, provides a reproducible, extensible testbed for future spatial audio encoder research. It also permits principled evaluation under perturbative analysis, complementing probe-based readout.

Limitations include the controlled, single-source synthetic environment, and the reliance on linear decodability as sole assessment. Future work is suggested along two axes: (1) extending SARL to multi-source, noisy, or real-world acoustics, and (2) exploring richer probing mechanisms (non-linear, causal interventions) and multi-view self-supervised objectives capable of richer room representation learning.

Conclusion

This work establishes a systematic, controlled approach for evaluating spatial factor encoding in pretrained audio representations, uncovering persistent representational gaps — chiefly the poor encoding of room factors relative to source characteristics — across state-of-the-art encoder paradigms. The SARL benchmark provides a robust platform for diagnosing, comparing, and ultimately improving spatial audio models. Progress toward spatially comprehensive audio foundation models will require not only architectural and supervisory innovation but also targeted benchmarks such as SARL to meaningfully quantify advancements.