Symbolic Music Diffusion with Mamba (SMDIM)

• The paper demonstrates that SMDIM achieves near-linear complexity and efficient long sequence modeling by integrating Structured State Space Models with the Mamba-FeedForward-Attention block.
• This model employs a hierarchical pipeline with discrete diffusion processes, enabling robust unconditional generation and precise conditional infilling of symbolic music tokens.
• Empirical validations on diverse datasets, including FolkDB, show superior attribute alignment and computational efficiency compared to transformer-based benchmarks.

Symbolic Music Diffusion with Mamba (SMDIM) is a diffusion-based generative model architecture for symbolic music that integrates Structured State Space Models (SSMs) and the Mamba-FeedForward-Attention (MFA) block, balancing scalability for long sequences with high musical expressiveness and detailed local structure. SMDIM evolves the diffusion paradigm for symbolic music by replacing transformer-centric architectures with SSM-based modules, achieving near-linear computational complexity and efficient modeling of long-range dependencies, as demonstrated on diverse datasets, including the FolkDB collection of traditional Chinese folk music (Yuan et al., 27 Jul 2025).

1. Hybrid Architecture: Diffusion, SSMs, and the MFA Block

SMDIM is built around a hierarchical pipeline: a discrete music sequence (using the REMI representation) is embedded by a shared layer, followed by a 1D convolution that both compresses the sequence and increases the feature dimension to encode local temporal details. The backbone consists of multiple stacked MFA blocks, each containing three components in sequence:

• Mamba Layer: Implements a selective structured state space model for global dependency modeling with linear complexity in sequence length. The state update equations are:

$x' = \sigma(\text{Conv1D}(\text{Linear}(x)))$

$z = \sigma(\text{Linear}(x))$

$$y' = \text{Linear}(\text{SelectiveSSM}(x') \odot z)$$

$y = \text{LayerNorm}(y' + x)$

The SelectiveSSM module updates hidden states as $h_t = A h_{t-1} + B x'_t$, using parameterized matrices $A$, $B$, and $C$ that are optimized during training.

• FeedForward Layer: A non-linear transformation inspired by transformer feedforward blocks further refines feature representations.
• Self-Attention Layer: Preserves fine-grained token-level precision; used sparingly to control computational cost, but provides fine local detail critical for precise musical events.

This architecture enables SMDIM to efficiently model long sequences by leveraging global SSM-based modeling and inserting self-attention "patches" for high-resolution local correction.

2. Computational Efficiency and Scalability

SMDIM is designed to address the prohibitive quadratic complexity of transformer self-attention when modeling long musical sequences. By relying primarily on the Mamba (SSM) and feedforward layers (both linear in sequence length $L$), and minimizing the use of self-attention (quadratic in $L$), the MFA block achieves the following complexity:

$\Omega(\text{MFA}) = 8LDN + LD^2 + L^2 D$

where $L$ is sequence length, $D$ is feature dimension, and $N$ is SSM state size.

Empirical results show SMDIM reducing GPU memory consumption by over 30% (21 GB vs 35 GB for a comparable transformer), and decreasing generation step time by 35% (0.35 s vs 0.54 s at $L=2048$), thereby enabling efficient modeling of long musical works (Yuan et al., 27 Jul 2025).

3. Diffusion Processing and Discrete Denoising

SMDIM employs a diffusion probabilistic model applied to sequences of symbolic music tokens converted via the REMI representation. The denoising process operates using a discrete DDPM framework:

• Forward process: Adds noise to the token sequence over multiple steps (using an absorbing state mechanism), yielding a sequence of increasingly corrupted representations.
• Reverse process: Proceeds through the MFA backbone and reconstructs the original symbolic sequence, stepwise denoising in parallel, thereby achieving non-autoregressive, globally consistent outputs.

This discrete denoising process is critical for both unconditional music generation and post-hoc conditional infilling without the exposure bias and sequential error accumulation of autoregressive models (Yuan et al., 27 Jul 2025).

4. Evaluation: Generation Quality and Expressiveness

SMDIM has been validated on MAESTRO, POP909, and FolkDB (comprising traditional Chinese folk music). Evaluation metrics include:

• Average Overlap Area (OA): Across pitch, note density, IOI (inter-onset interval), and velocity distributions, SMDIM demonstrated higher OA scores than transformer-based diffusion models.
• FolkDB performance: The model generated symbolic music that closely matched ground-truth attribute distributions in the FolkDB dataset, reflecting its ability to capture characteristic modal, rhythmic, and structural features found in non-Western musical corpora.
• Scalability metrics: SMDIM maintained computational efficiency during long-sequence generation, a property not achievable by quadratic-complexity transformer baselines.

These results establish SMDIM as state-of-the-art for both statistical and perceptual metrics when operating on long, structurally complex symbolic music sequences (Yuan et al., 27 Jul 2025).

5. Adaptability Beyond Symbolic Music

The architectural principles developed in SMDIM—hybrid SSM/self-attention modeling and efficient diffusion-based discrete generation—extend naturally to other sequence generation domains:

• Text Generation: The global coherence requirement in document or story generation is analogous to the need for musical form; SSM layers capture context efficiently, while self-attention patches insert necessary local detail.
• Time Series Forecasting: SMDIM’s scalable linear complexity suits high-dimensional, long-horizon time series, preserving both macro trends and micro fluctuations.
• Video and Audio Synthesis: Where temporal dependencies are long-range but require local, frame-level consistency, SMDIM's architecture can be directly repurposed.

This generality demonstrates the architectural flexibility and broad applicability of SMDIM principles in sequence modeling.

6. Key Innovations and Comparative Position

SMDIM introduces several architectural advances over previous symbolic music generation methods:

• MFA Block Design: The sequential arrangement—Mamba layer $\rightarrow$ FeedForward $\rightarrow$ Self-Attention—systematically compresses global structures, injects non-linearity, and re-establishes high-fidelity local structure, outperforming SSM-only and transformer-only baselines in both efficiency and output quality.
• Near-Linear Complexity: Achieved by restricting the use of quadratic-complexity modules without significant loss in expressive power.
• Diffusion-based Discrete Denoising: Integrates robust sequence refinement within the latent or token domain, facilitating both creative generation and constrained tasks like infilling.

In comparative analyses, SMDIM exhibits superior attribute alignment, sequence-level quality, and computational efficiency across datasets, including challenging non-Western corpora, marking a significant advance in scalable, expressive symbolic sequence modeling (Yuan et al., 27 Jul 2025).