---
title: 'SpaceLM: End-to-End State-Space Audio-Language Model'
url: https://www.emergentmind.com/topics/spacelm
type: topic
---

# SpaceLM: End-to-End State-Space Audio-Language Model

SpaceLM is the first end-to-end State-Space Large Audio-Language Model (ssLALM) that replaces all Transformer modules in standard Large Audio Language Model (LALM) pipelines with state-space model (SSM) components. By leveraging recent advances in SSM architectures, SpaceLM achieves competitive performance to Transformer-based models in audio classification and captioning tasks, while offering improved computational and memory efficiency due to the linear scaling properties of SSMs. Its architecture systematically replaces both the audio perception Transformer and the Transformer-based large language model (LLM, e.g., LLaMA) with structured SSM pipelines, primarily based on the S4 (selective state-space) paradigm [2411.15685].

## 1. SpaceLM Architecture

SpaceLM introduces a dual-SSM architecture for audio-language modeling by substituting both conventional audio Transformer front-ends and Transformer-based language models with SSM building blocks:

- **Audio Encoder (DASS):** The Distilled Audio State-Space Model (DASS) is a four-stage hierarchy of SSM blocks, each stage including intermediate 2× spatial downsampling. The input is a log-mel spectrogram of shape $(T=1024, F=128)$. After SSM processing and downsampling, the output is a feature map $(T'=32, F'=4, C=768)$, which is further projected via $3\times3$ convolution with stride 2 followed by a linear projection to dimension $d_\text{embed} = 2560$.
- **Language Model (Mamba SSM LLM):** The language backbone uses the Mamba SSM LLM with 2.8B parameters, replacing all Transformer self-attention and feedforward blocks with S4 layers. The model is pretrained on The Pile and subsequently frozen.
- **Trainable Adapters (LoRA):** Low-Rank Adaptation (LoRA) adapters (rank 8, scaling $\alpha=16$) are inserted into each S4 input-projection layer, enabling efficient fine-tuning with 43M (small) or 62M (medium) total trainable parameters.
- **Token Flow:** The final audio embedding (from DASS) is concatenated as a one-token prefix to the token sequence before being processed by the sequence of S4 blocks in the SSM LLM for autoregressive text decoding.

## 2. State-Space Modeling and S4 Formulation

SpaceLM’s computational backbone is the S4 state-space model, following a continuous-to-discrete mapping:

- **Continuous SSM:**
  $$
  \dot{h}(t) = A h(t) + B x(t)
  $$
  $$
  y(t) = C h(t)
  $$
- **Discretized (Zero-Order Hold):**
  $$
  h_t = \bar{A} h_{t-1} + \bar{B} x_t,\quad y_t = C h_t
  $$
  with $\bar{A} = \exp(\Delta A)$ and appropriate $\bar{B}$.
- **Convolutional View:**
  $$
  y_t = \sum_{i=0}^t K_i x_{t-i},\quad K_i = C \bar{A}^i \bar{B}
  $$
All SSM layers in SpaceLM (both encoder and LLM) implement these dynamics, with learnable $A$, $B$, $C$ parameters and $D=0$. The SSM formulation is implemented such that parallel training is compatible with standard pipelines [2411.15685].

## 3. Implementation Details and Training Paradigm

- **DASS Audio Encoder:** Composed of four SSM stages, each with $N=768$ hidden units and $2\times2$ average pooling (in groups 1–3). Final features are globally pooled, then projected to the dimension expected by the SSM LLM.
- **SSM LLM (“Mamba”):** Model width $d_\text{model}=2560$, depth $L\approx32$ (S4 blocks), hidden S4 state $N\approx512$. Pretrained weights are frozen and LoRA adapters are the only trainable path in the LLM.
- **Audio-Text Conditioning:** The DASS embedding is treated as a prefix conditioning token in the language model context.
- **Parameter Regime:** The small variant has $2.85$B total parameters (2.8B SSM LLM, 47M DASS), with trainable LoRA adapters and DASS heads accounting for approximately 1.5% of total parameters.

## 4. Computational and Memory Efficiency

SpaceLM demonstrates key computational advantages over Transformer-based LALMs:

| Model               | Total Parameters | Trainable Parameters | Sequence Complexity | Memory Complexity        |
|---------------------|-----------------|---------------------|---------------------|-------------------------|
| Small SpaceLM       | 2.85B           | 43M                 | $O(L d_\text{model} N)$ | $O(L d_\text{model})$    |
| Hybrid AST+LLaMA    | 6.8B            | 42M                 | $O(L^2 d_\text{model})$ | $O(L^2 d_\text{model})$  |

Time and memory per layer in SpaceLM both scale linearly with sequence length $L$, as opposed to the quadratic scaling of self-attention. This efficiency is achieved without sacrificing competitive downstream task performance [2411.15685].

## 5. Evaluation and Empirical Results

Evaluations on eight closed-ended audio classification tasks and two audio captioning benchmarks demonstrate that SpaceLM achieves or surpasses Transformer baselines:

| Model            | Cls-AVG | Cap-AVG | Notable Results                                                  |
|------------------|---------|---------|------------------------------------------------------------------|
| LTU (7B, trans.) | 50.3    | 14.5    | 83.1 (ESC50), 45.9 (DCASE), 17.0 (AudioCaps), 11.9 (Clotho)     |
| Small SpaceLM    | 50.0    | 15.1    | 84.3 (ESC50), 46.4 (DCASE), 18.0 (AudioCaps), 12.1 (Clotho)     |
| Medium SpaceLM   | 51.4    | 14.7    | 86.8 (ESC50), 47.9 (DCASE), 17.7 (AudioCaps), 11.7 (Clotho)     |

Medium SpaceLM matches or exceeds the LTU on overall classification (51.4% vs. 50.3%) and achieves comparable SPICE for captioning (14.7 vs. 14.5) with less than half the total parameters and strictly linear scaling [2411.15685].

## 6. Architectural Significance and Research Context

SpaceLM advances the SSM approach in two novel aspects: (1) full replacement of both audio perception and language backbone by SSM blocks; (2) demonstration that linear-scaling SSM LALMs can perform competitively with parameter- and computation-intensive Transformer models. The architecture leverages S4-based SSMs for both feature hierarchy extraction (via DASS) and autoregressive lexical modeling (“Mamba” SSM LLM). LoRA adapters enable sample-efficient fine-tuning, making the framework practical for memory- and time-constrained settings.

This approach distinguishes SpaceLM from earlier attempts that only replaced the audio (front-end) Transformer with SSM but retained Transformer LLMs. No prior work implemented an end-to-end SSM pipeline for LALM tasks before SpaceLM [2411.15685].

## 7. Future Directions and Limitations

SpaceLM’s SSM paradigm presents a foundation for further research in efficient multi-modal sequence modeling. Primary limitations include the restriction to closed-ended and captioning tasks; the impact of SSM-based LALMs on more complex generative and interactive tasks remains to be established. Moreover, while all results are achieved with frozen language model weights and LoRA adapters, potential exists for more extensive end-to-end optimization or for exploring larger SSM language models. A plausible implication is that further scaling and domain adaptation of SSM-based LLMs may yield additional gains in both accuracy and efficiency.

SpaceLM provides a reference implementation for end-to-end state-space models in audio-language tasks and demonstrates empirically the feasibility of moving beyond Transformer architectures without a loss of performance in standard benchmarks [2411.15685].

Source: https://www.emergentmind.com/topics/spacelm