Spatial Audio Language Model (SALM)

• Spatial Audio Language Model (SALM) is a multimodal framework that captures and fuses semantic and spatial audio features using dedicated text and audio encoders.
• It employs multi-modal contrastive learning with specialized DOA loss to align audio and text representations, achieving near-perfect zero-shot spatial classification.
• SALM enables text-driven spatial audio editing and retrieval, providing practical solutions for spatial scene analysis in AR/VR and intelligent systems.

A Spatial Audio LLM (SALM) is a multimodal machine learning framework that explicitly captures, aligns, and manipulates both the semantic and spatial attributes of audio in conjunction with natural language. Recent research positions SALM as a critical advancement for spatial audio understanding, retrieval, generation, and editing, with direct applications in spatial scene analysis, AR/VR, and human-centric intelligent systems (Hu et al., 22 Jul 2025).

1. Framework and Core Methodology

SALM introduces a dual-encoder architecture, where spatial sound is decomposed into semantic and spatial components through structured audio embeddings (Hu et al., 22 Jul 2025). The model consists of:

• Text Encoder: Based on the RoBERTa architecture adapted from LAION‑CLAP, it encodes both standard and spatially enhanced captions, producing Text Embeddings (E_T) and Spatial Text Embeddings (E_ST).
• Audio Encoder: Comprised of two branches:
  • Audio Semantic Branch: Processes the omnidirectional channel of First-Order Ambisonics (FOA) input, extracting semantic content.
  • Audio Spatial Branch: Processes all FOA channels, extracting directional cues such as direction-of-arrival (DOA).

The outputs are Audio Semantic Embeddings (E_ASe) and Audio Spatial Embeddings (E_ASp), which are fused using a learnable vector s to produce Joint Audio Embeddings (E_JA):

$E_{JA} = E_{ASe} + s \odot E_{ASp}$

where $\odot$ denotes element-wise multiplication, enabling SALM to disentangle or combine semantic and spatial content as needed.

2. Multi-modal Contrastive Learning and Loss Functions

SALM employs multi-modal contrastive learning to align audio and text representations at both the semantic and spatial levels:

• Semantic Alignment: A contrastive loss $\mathcal{L}_{CL}$ aligns Audio Semantic Embeddings with Text Embeddings.
• Spatial Alignment: A spatial contrastive loss $\mathcal{L}_{sCL}$ aligns Joint Audio Embeddings with Spatial Text Embeddings. Training pairs positive (matching) and negative (mismatched) audio-text or audio-spatial-text samples within each batch.
• Direction-of-Arrival Supervision: A specialized DOA loss, $\mathcal{L}_{DOA}$, directly supervises the Audio Spatial Branch by minimizing the cosine distance between the predicted and ground-truth DOA vectors.

The total loss is formalized as:

$$\mathcal{L}_{SALM} = \frac{1}{2} (\mathcal{L}_{CL} + \mathcal{L}_{sCL}) + \mathcal{L}_{DOA}$$

This approach ensures that the learned embeddings are both robustly cross-modal and finely sensitive to spatial cues (Hu et al., 22 Jul 2025).

3. Structured Audio Embeddings and Cross-modal Manipulation

A central feature of SALM is its explicit separation and fusion of spatial and semantic information:

• Decoupling: The architecture enables discrete extraction of semantic (E_ASe) and spatial (E_ASp) features, facilitating targeted optimization and transfer.
• Fusion for Joint Understanding: By aggregating these embeddings, the model forms E_JA, supporting queries and retrieval tasks that require both what and where information.
• Cross-modal Operations: Embeddings permit direction swapping via replacement of E_ASp with a spatial text embedding (E_TDi) while preserving the norm, producing an edited Joint Audio Embedding:

$$\tilde{E}_{JA} = E_{ASe} + s \odot \left( \frac{\|E_{ASp}\| \cdot E_{TDi}}{\|E_{TDi}\|} \right)$$

This operation allows for text-driven spatial audio editing, such as relocating the perceived sound source through natural language input while holding the audio identity invariant.

4. Zero-Shot Direction Classification and Editing Capabilities

SALM achieves near-perfect zero-shot spatial classification accuracy, enabled by its structured embeddings:

• Procedure: The Audio Spatial Embedding (or E_JA) is compared via cosine similarity to a set of text embeddings corresponding to canonical direction descriptions (e.g., "The sound is coming from the south").
• Performance: Evaluation across eight principal directions yields 99.9–100% accuracy for both E_ASp and E_JA, demonstrating robust generalization to unseen spatial queries (Hu et al., 22 Jul 2025).
• Editing: Experiments confirm that replacing the spatial component with a new directional text embedding minimally impacts the semantic fidelity in retrieval tasks, validating both the modularity and precision of the embedding space.

5. Experimental Results and Benchmarks

Empirical evaluation on established spatial audio benchmarks highlights SALM's superior performance:

Model	sClotho R@1	sClotho Loc. Error (°)	sAudioCaps R@1	sAudioCaps Loc. Error (°)
SALM	0.437	1.92	0.443	2.02
SALM-s	0.430	2.01	0.439	2.16
LAION-CLAP	0.234	64.56	0.247	62.07

• Retrieval Accuracy: SALM nearly doubles the recall@1 relative to non-spatial baselines.
• Localization: The localization error is reduced to approximately 2 degrees, an order of magnitude improvement over methods lacking explicit spatial mechanisms.
• Editing Consistency: The spatial editing mechanism does not degrade overall retrieval or localization accuracy (Hu et al., 22 Jul 2025).

6. Applications and Research Implications

The structured design and operational flexibility of SALM enable a range of spatial audio understanding, manipulation, and retrieval applications:

• Spatial Querying: Enables language-driven localization—e.g., querying for "dog barking from the left" retrieves audio where both the event and its direction match the description.
• Editing and Augmentation: Direct modification of spatial attributes through textual input permits spatial remapping of sounds, relevant for AR/VR, scene composition, and game audio design.
• Zero-shot Generalization: The zero-shot spatial classification ability positions SALM for deployment in settings where explicit class labels or spatial annotations are unavailable.
• Benchmarking: The dual-branch contrastive approach and explicit loss functions provide benchmarks and architectural templates for future multimodal models addressing spatial audio tasks.

7. Limitations and Outlook

While SALM advances spatial audio-language alignment, several challenges and potential directions are noted:

• Scalability to Higher-Order Ambisonics: Current implementation is tailored to FOA. Extension to higher spatial orders remains a topic for further work.
• Real-world Generalization: While empirical results on simulated data are strong, performance across diverse real-world scenes is not yet explored at scale.
• Fusion Strategies: The weighted sum fusion of semantic and spatial embeddings is a practical solution, but further research may investigate adaptive or attention-based combination mechanisms to optimize downstream task performance.

SALM provides a robust, modular, and extensible model for cross-modal spatial audio-language processing, establishing a technical foundation for a wide class of spatially aware audio-language systems (Hu et al., 22 Jul 2025).