SpaceLM: End-to-End State-Space Audio-Language Model

• SpaceLM is the first end-to-end state-space audio-language model that replaces Transformer modules with SSM components for enhanced efficiency.
• It integrates a dual architecture with the DASS audio encoder and the Mamba SSM LLM, using LoRA adapters for efficient fine-tuning.
• Empirical evaluations reveal competitive or superior performance in audio classification and captioning, with linear computational and memory scaling.

SpaceLM is the first end-to-end State-Space Large Audio-LLM (ssLALM) that replaces all Transformer modules in standard Large Audio LLM (LALM) pipelines with state-space model (SSM) components. By leveraging recent advances in SSM architectures, SpaceLM achieves competitive performance to Transformer-based models in audio classification and captioning tasks, while offering improved computational and memory efficiency due to the linear scaling properties of SSMs. Its architecture systematically replaces both the audio perception Transformer and the Transformer-based LLM (LLM, e.g., LLaMA) with structured SSM pipelines, primarily based on the S4 (selective state-space) paradigm (Bhati et al., 2024).

1. SpaceLM Architecture

SpaceLM introduces a dual-SSM architecture for audio-language modeling by substituting both conventional audio Transformer front-ends and Transformer-based LLMs with SSM building blocks:

• Audio Encoder (DASS): The Distilled Audio State-Space Model (DASS) is a four-stage hierarchy of SSM blocks, each stage including intermediate 2× spatial downsampling. The input is a log-mel spectrogram of shape $(T=1024, F=128)$. After SSM processing and downsampling, the output is a feature map $(T'=32, F'=4, C=768)$, which is further projected via $3\times3$ convolution with stride 2 followed by a linear projection to dimension $d_\text{embed} = 2560$.
• LLM (Mamba SSM LLM): The language backbone uses the Mamba SSM LLM with 2.8B parameters, replacing all Transformer self-attention and feedforward blocks with S4 layers. The model is pretrained on The Pile and subsequently frozen.
• Trainable Adapters (LoRA): Low-Rank Adaptation (LoRA) adapters (rank 8, scaling $\alpha=16$) are inserted into each S4 input-projection layer, enabling efficient fine-tuning with 43M (small) or 62M (medium) total trainable parameters.
• Token Flow: The final audio embedding (from DASS) is concatenated as a one-token prefix to the token sequence before being processed by the sequence of S4 blocks in the SSM LLM for autoregressive text decoding.

2. State-Space Modeling and S4 Formulation

SpaceLM’s computational backbone is the S4 state-space model, following a continuous-to-discrete mapping:

• Continuous SSM:

$\dot{h}(t) = A h(t) + B x(t)$

$y(t) = C h(t)$

• Discretized (Zero-Order Hold):

$$h_t = \bar{A} h_{t-1} + \bar{B} x_t,\quad y_t = C h_t$$

with $\bar{A} = \exp(\Delta A)$ and appropriate $\bar{B}$.

• Convolutional View:

$(T'=32, F'=4, C=768)$0

All SSM layers in SpaceLM (both encoder and LLM) implement these dynamics, with learnable $(T'=32, F'=4, C=768)$1, $(T'=32, F'=4, C=768)$2, $(T'=32, F'=4, C=768)$3 parameters and $(T'=32, F'=4, C=768)$4. The SSM formulation is implemented such that parallel training is compatible with standard pipelines (Bhati et al., 2024).

3. Implementation Details and Training Paradigm

• DASS Audio Encoder: Composed of four SSM stages, each with $(T'=32, F'=4, C=768)$5 hidden units and $(T'=32, F'=4, C=768)$6 average pooling (in groups 1–3). Final features are globally pooled, then projected to the dimension expected by the SSM LLM.
• SSM LLM (“Mamba”): Model width $(T'=32, F'=4, C=768)$7, depth $(T'=32, F'=4, C=768)$8 (S4 blocks), hidden S4 state $(T'=32, F'=4, C=768)$9. Pretrained weights are frozen and LoRA adapters are the only trainable path in the LLM.
• Audio-Text Conditioning: The DASS embedding is treated as a prefix conditioning token in the LLM context.
• Parameter Regime: The small variant has $3\times3$0B total parameters (2.8B SSM LLM, 47M DASS), with trainable LoRA adapters and DASS heads accounting for approximately 1.5% of total parameters.

4. Computational and Memory Efficiency

SpaceLM demonstrates key computational advantages over Transformer-based LALMs:

Model	Total Parameters	Trainable Parameters	Sequence Complexity	Memory Complexity
Small SpaceLM	2.85B	43M	$3\times3$1	$3\times3$2
Hybrid AST+LLaMA	6.8B	42M	$3\times3$3	$3\times3$4

Time and memory per layer in SpaceLM both scale linearly with sequence length $3\times3$5, as opposed to the quadratic scaling of self-attention. This efficiency is achieved without sacrificing competitive downstream task performance (Bhati et al., 2024).

5. Evaluation and Empirical Results

Evaluations on eight closed-ended audio classification tasks and two audio captioning benchmarks demonstrate that SpaceLM achieves or surpasses Transformer baselines:

Model	Cls-AVG	Cap-AVG	Notable Results
LTU (7B, trans.)	50.3	14.5	83.1 (ESC50), 45.9 (DCASE), 17.0 (AudioCaps), 11.9 (Clotho)
Small SpaceLM	50.0	15.1	84.3 (ESC50), 46.4 (DCASE), 18.0 (AudioCaps), 12.1 (Clotho)
Medium SpaceLM	51.4	14.7	86.8 (ESC50), 47.9 (DCASE), 17.7 (AudioCaps), 11.7 (Clotho)

Medium SpaceLM matches or exceeds the LTU on overall classification (51.4% vs. 50.3%) and achieves comparable SPICE for captioning (14.7 vs. 14.5) with less than half the total parameters and strictly linear scaling (Bhati et al., 2024).

6. Architectural Significance and Research Context

SpaceLM advances the SSM approach in two novel aspects: (1) full replacement of both audio perception and language backbone by SSM blocks; (2) demonstration that linear-scaling SSM LALMs can perform competitively with parameter- and computation-intensive Transformer models. The architecture leverages S4-based SSMs for both feature hierarchy extraction (via DASS) and autoregressive lexical modeling (“Mamba” SSM LLM). LoRA adapters enable sample-efficient fine-tuning, making the framework practical for memory- and time-constrained settings.

This approach distinguishes SpaceLM from earlier attempts that only replaced the audio (front-end) Transformer with SSM but retained Transformer LLMs. No prior work implemented an end-to-end SSM pipeline for LALM tasks before SpaceLM (Bhati et al., 2024).

7. Future Directions and Limitations

SpaceLM’s SSM paradigm presents a foundation for further research in efficient multi-modal sequence modeling. Primary limitations include the restriction to closed-ended and captioning tasks; the impact of SSM-based LALMs on more complex generative and interactive tasks remains to be established. Moreover, while all results are achieved with frozen LLM weights and LoRA adapters, potential exists for more extensive end-to-end optimization or for exploring larger SSM LLMs. A plausible implication is that further scaling and domain adaptation of SSM-based LLMs may yield additional gains in both accuracy and efficiency.

SpaceLM provides a reference implementation for end-to-end state-space models in audio-language tasks and demonstrates empirically the feasibility of moving beyond Transformer architectures without a loss of performance in standard benchmarks (Bhati et al., 2024).