Spatial-Frequency Mamba Blocks

• Spatial-Frequency Mamba Blocks are modules that fuse spatial and frequency representations using dual branches, enabling enhanced multi-scale modeling.
• They mitigate Mamba’s order sensitivity by combining wavelet or Fourier-based frequency decompositions with tailored scanning orders, ensuring robust data processing.
• These blocks employ cross-attention fusion to balance local detail with global context, leading to improved efficiency in image generation and reconstruction.

Spatial-Frequency Mamba Blocks are Mamba-based state-space modules that couple spatial sequence modeling with an explicit frequency-oriented representation. In the narrow sense, the term is most directly associated with DiMSUM, where a latent feature map $x \in \mathbb{R}^{C \times H \times W}$ is processed in parallel by a spatial Mamba branch and a wavelet-frequency branch, and the two outputs are fused by query-swapped cross-attention (Phung et al., 2024). In broader usage, closely related designs appear under names such as time-frequency, spectral-spatial, and spatio-frequency Mamba, depending on whether the complementary representation is a wavelet decomposition, Fourier amplitude/phase, STFT axes, or hyperspectral bands. Across these variants, the common objective is to mitigate the order sensitivity of Mamba-style scanning while preserving both local detail and long-range structure (Xiao et al., 2024, Huang et al., 2024, Zou et al., 2024).

1. Conceptual basis and motivation

Spatial-Frequency Mamba Blocks arise from a recurrent observation in the literature: standard Mamba is strong at long-sequence modeling, but its behavior on structured visual or multimodal data depends heavily on how multidimensional inputs are serialized. In DiMSUM, this issue is described as especially limiting for image generation, because any single 2D scan exposes only one linear ordering of the image and therefore imposes a weak, manually chosen inductive bias (Phung et al., 2024). Related papers make analogous arguments in other settings: 3D medical segmentation emphasizes that naive serialization does not naturally preserve volumetric structure, hyperspectral classification emphasizes redundancy and the need to model spatial and spectral dependencies jointly, and multimodal MRI reconstruction emphasizes that purely spatial fusion misses global Fourier-domain properties (Zhang et al., 5 Aug 2025, He et al., 2024, Zou et al., 2024).

The “frequency” component is not uniform across tasks. In image generation and restoration it often denotes wavelet or Fourier structure; in sound source localization it denotes the STFT frequency axis; in hyperspectral imaging it denotes the spectral-band dimension; and in some video models it is refined further into amplitude- and phase-specific motion cues (Xiao et al., 2024, Huang et al., 2024, Li et al., 31 Jul 2025). This variation does not erase the family resemblance. The recurring pattern is to let Mamba operate on more than one ordering of the same signal, typically one preserving spatial layout and another exposing global, bandwise, or subband structure that a single raster-like scan does not express well.

A second motivation is computational. Several papers frame Mamba as a middle ground between CNNs and Transformers: CNNs are local, while Transformers model global relations at quadratic cost; Mamba offers long-range dependency modeling with linear complexity, but needs better structural priors once inputs stop being naturally one-dimensional (Zou et al., 2024, He et al., 2024, Zhang et al., 5 Aug 2025). Spatial-frequency blocks can therefore be understood as attempts to retain Mamba’s linear-time sequence modeling while repairing its domain mismatch.

2. DiMSUM as a representative spatial-frequency block

DiMSUM provides one of the most explicit formulations of a Spatial-Frequency Mamba Block. The model operates in latent space, as in LDM. The appendix configuration used in experiments specifies depth 16, hidden size 1024, patch size 2, and learnable absolute positional embeddings; attention is inserted every $k=4$ layers, and the attention weights are shared globally across the transformer insertions (Phung et al., 2024).

At block level, the input feature map $x \in \mathbb{R}^{C \times H \times W}$ is sent through two parallel paths. The spatial path applies Mamba with a chosen 2D scan order and produces $f_s$. The frequency path first applies a Haar wavelet decomposition, then performs Mamba-style scanning on the resulting subbands, and produces $f_w$. The two outputs are fused into $f_{\text{out}}$ by a cross-attention fusion layer: $$q_s, k_s, v_s = \mathrm{Linear}(f_s), \qquad q_w, k_w, v_w = \mathrm{Linear}(f_w),$$

$$f_{\text{out}} = \mathrm{Linear}\Big(\mathrm{Concat}\big(\mathrm{Attn}(q_s, k_w, v_w),\; \mathrm{Attn}(q_w, k_s, v_s)\big)\Big).$$

The wavelet path is central rather than auxiliary. DiMSUM constructs four subbands

$$\{x_{LL}, x_{LH}, x_{HL}, x_{HH}\}, \qquad x_* \in \mathbb{R}^{H/2 \times W/2},$$

corresponding to the low-frequency approximation and three directional high-frequency components. Because the wavelet filters are pairwise orthogonal, the transform is invertible and the inverse discrete wavelet transform recovers the original resolution. In the main setting, the block uses a two-level Haar transform on encoded features, and it decomposes every wavelet subband rather than only the $LL$ band. The paper emphasizes that this “evenly separates” the input into multiple wavelet patches and preserves information across the whole frequency spectrum (Phung et al., 2024).

DiMSUM also modifies the Mamba recurrence for conditioning. Instead of initializing the hidden state to zero, it uses

$k=4$0

where $k=4$1 is a conditioning embedding. For unconditional generation, $k=4$2 is a learned token; for class-conditional generation, $k=4$3 is a class embedding. This “Conditional Mamba” is enabled by default. The resulting design is not merely a different scan pattern. It is a dual-domain block in which spatial order modeling, wavelet decomposition, cross-domain alignment, and conditioning are jointly defined inside the same recurrent backbone (Phung et al., 2024).

3. Scanning order, tokenization, and frequency parameterization

A defining trait of spatial-frequency Mamba research is that the choice of scan order is treated as a first-class design variable. In DiMSUM, the default spatial scan is Sweep-4 with interleaved scanning order, and the paper benchmarks Bi, Sweep-4, Sweep-8, Zigzag-8, JPEG-8, and Window scans. Sweep-4 is best for the spatial branch, while the wavelet branch performs better with window scanning than with a standard Sweep-4 applied to wavelet tokens. The paper explicitly notes that more directions do not necessarily help, and that the best configuration is a deliberate separation: sweep-based ordering in the spatial path and local window scanning in wavelet space (Phung et al., 2024).

The wavelet branch in DiMSUM also changes what a “window” means. After two-level decomposition, the subbands are concatenated into a 1D sequence and processed with a LocalMamba-inspired scan that uses two sliding directions, left-to-right and top-to-bottom. Reverse directions are avoided because low-frequency subbands should be input first, so the frequency ordering is asymmetric. The figure description further specifies a $k=4$4 window scanning across multiple wavelet subbands “like the CNN kernel.” This is a shift from scanning pixels in the image plane to scanning tokens that have already been reorganized by frequency.

Comparable scan redesigns appear in other domains. SSFMamba introduces a 3D multi-directional scanning mechanism in the frequency branch, splitting the transformed volume into in-slice ordering, cross-slice ordering, and 3D local ordering, explicitly to respect conjugate symmetry and volumetric structure (Zhang et al., 5 Aug 2025). IGroupSS-Mamba partitions hyperspectral features into four non-overlapping interval groups and assigns them left-to-right, right-to-left, top-to-bottom, and bottom-to-top scanning directions, arguing that this yields non-redundant sequence modeling (He et al., 2024). Vcamba replaces generic flattening in the frequency domain with spiral low-to-high and high-to-low sequential scanning from the FFT center outward, to preserve semantic continuity among neighboring frequency components (Li et al., 31 Jul 2025). FA-Mamba uses different paths for different wavelet sub-bands, scanning $k=4$5, $k=4$6, and $k=4$7 horizontally and vertically, but scanning $k=4$8 diagonally, with traversal adjusted to sub-band texture distributions (Pan et al., 3 Dec 2025).

This body of work suggests that scan-order engineering is inseparable from the notion of a spatial-frequency block. The frequency transform reorganizes information, but the sequence model still depends on an ordering; consequently, most successful designs specify not only what representation is used, but also how that representation is traversed.

4. Fusion mechanisms and global compensation

The interaction between spatial and frequency representations is usually explicit rather than incidental. DiMSUM states that simple concatenation is too weak because spatial and frequency features are not well aligned; in ablation on CelebA-HQ 256, naive concatenation with Wavelet Mamba yields FID 5.87, whereas cross-attention fusion improves this to 4.92, and the swapped-query variant outperforms simple linear projection and plain attention-based fusion (Phung et al., 2024). The conceptual point is precise: each branch queries the other branch’s keys and values, so spatial features request frequency-sensitive evidence while wavelet features request order- and layout-sensitive context.

DiMSUM further adds a globally shared transformer block after every four DiM blocks. The same transformer weights are reused across these positions, reducing parameter overhead. In the reported ablation, adding this shared transformer improves FID from 4.92 to 4.65 on CelebA-HQ 256 and raises recall from 0.50 to 0.52. The paper positions this as compensation for Mamba’s residual order dependence, because transformer attention captures global relationships without relying on a manually chosen scan order (Phung et al., 2024).

Other spatial-frequency Mamba models adopt different but related fusion principles. MMR-Mamba combines a Target modality-guided Cross Mamba in the spatial domain, a Selective Frequency Fusion module in the Fourier domain, and an Adaptive Spatial-Frequency Fusion module that supplements less informative channels in one domain with the corresponding channels from the other. The spatial branch uses the target modality as a gate, the frequency branch selectively harmonizes amplitude while fusing phase, and the final cross-domain fusion uses BN scaling parameters to decide which channels should be reinforced (Zou et al., 2024). Vcamba’s SFMF module performs both sequence-to-sequence concatenation and point-to-point concatenation, aiming to preserve global cross-domain conditioning and local spatial correspondence simultaneously (Li et al., 31 Jul 2025). SSFMamba, by contrast, uses Mamba as a domain-specific encoder in each branch and then merges spatial and frequency features through an MLP after summation, with a residual addition back to the input (Zhang et al., 5 Aug 2025).

A common misconception is that a spatial-frequency Mamba block is simply a Mamba branch plus extra frequency channels. The reported designs contradict that interpretation. Fusion is typically query-swapped, gated, channel-adaptive, or otherwise structured; the central design problem is not only how to extract dual-domain features, but how to reconcile their differing geometries and semantics.

5. Domain-specific variants

The literature uses closely related spatial-frequency constructions across a wide range of tasks. The specific meaning of “frequency” changes with modality, but the design pattern of complementary sequence views persists.

Domain	Representative design	Frequency notion
Image generation	DiMSUM (Phung et al., 2024)	Two-level Haar wavelet subbands fused with spatial Mamba
Image deraining	FreqMamba (Zhen et al., 2024)	2-layer wavelet packet sub-bands plus Fourier global modeling
Multi-modal MRI reconstruction	MMR-Mamba (Zou et al., 2024)	Fourier amplitude/phase selective fusion with spatial Cross Mamba
Sound source localization	TF-Mamba (Xiao et al., 2024)	Time-wise and frequency-wise BiMamba on STFT features
Hyperspectral classification	SS-Mamba (Huang et al., 2024); IGroupSS-Mamba (He et al., 2024)	Spectral-band sequences paired with spatial sequences
3D medical segmentation	SSFMamba (Zhang et al., 5 Aug 2025)	FFT magnitude/phase with symmetry-aware 3D directional scanning

FreqMamba is a particularly clear three-branch extension. Its Frequency-SSM Block contains spatial Mamba, frequency-band Mamba based on a 2-layer wavelet packet transform, and Fourier global modeling; the authors describe the frequency-band branch as a transition between Mamba and Fourier reasoning, and the Fourier branch as the “panoramic/global expert” for degradation modeling (Zhen et al., 2024). DemMamba extends the idea to raw video demoiréing by alternating Spatial Mamba Blocks and Temporal Mamba Blocks; the spatial block includes an Adaptive Frequency Block that applies FFT, a learnable frequency compressor, and inverse FFT to suppress moiré-related frequencies (Xu et al., 2024). FA-Mamba uses DWT/IWT, a Dual-Branch Feature Extraction Block with CNN and Mamba branches, a Prior-Guided Block for high-frequency residual learning, and an Adaptive Frequency Scanning Mechanism that changes scan geometry by sub-band type (Pan et al., 3 Dec 2025). In multimodal image fusion, SFMFusion defines a Spatial-Frequency Enhanced Mamba Block with three parallel branches—Mixed-scale Mamba Block, Channel Enhanced Block, and Frequency Enhanced Block—to address multi-scale spatial perception, inter-channel dependency, and explicit amplitude/phase modeling within a single feature extractor (Sun et al., 10 Nov 2025).

Not every hierarchical or multi-scale Mamba design belongs to this category in the strict sense. Hi-Mamba, for example, introduces local and region SSM branches and direction alternation for image super-resolution, but the paper explicitly states that it does not introduce explicit frequency-domain processing, Fourier transforms, or frequency-aware blocks in the strict sense (Qiao et al., 2024). This contrast clarifies a useful boundary: a spatial-frequency Mamba block typically includes an explicit secondary representation beyond pure spatial hierarchy.

6. Empirical record, misconceptions, and research directions

Ablation studies consistently attribute measurable gains to the spatial-frequency formulation rather than to Mamba alone. In DiMSUM on CelebA-HQ 256, a baseline conditional-Mamba model reports FID 6.19; adding Conditional Mamba improves to 5.27; adding Wavelet Mamba with concatenation worsens to 5.87; replacing concatenation with cross-attention improves to 4.92; and adding the shared transformer further improves to 4.65. The same study reports that wavelet outperforms DCT and EinFFT, and that two wavelet levels work best, whereas three levels degrade because the subbands become too small for the $k=4$9 latent patch grid (Phung et al., 2024).

The same pattern appears in restoration and reconstruction. On Rain100L, FreqMamba reports 39.18 / 0.9814 for the full model; removing the Fourier branch reduces PSNR by about 0.33 dB, removing the frequency-band branch reduces performance by about 0.1 dB, and removing the attention map also lowers results, indicating that global Fourier modeling, intermediate sub-band reasoning, and degradation-aware modulation all contribute (Zhen et al., 2024). In BraTS $x \in \mathbb{R}^{C \times H \times W}$0 reconstruction, MMR-Mamba reports a progression from 38.45 dB for the baseline to 39.05 dB with TCM, 40.49 dB with SFF, 40.66 dB with TCM + SFF, and 40.98 dB for the full model with ASFF, showing that spatial-domain selective fusion and Fourier-domain selective fusion are complementary rather than interchangeable (Zou et al., 2024). In SSFMamba on BraTS2023, the baseline gives Avg Dice 91.40 and Avg HD95 3.77, while the full model reaches Avg Dice 93.13 and Avg HD95 3.26, outperforming variants that use only the frequency-domain branch or only multi-directional scanning (Zhang et al., 5 Aug 2025).

Several misconceptions can therefore be addressed directly. First, spatial-frequency Mamba is not equivalent to “more scan directions.” DiMSUM explicitly reports that more directions do not necessarily help, and that scan strategies should differ across branches (Phung et al., 2024). Second, it is not tied to one transform family: wavelets, FFT amplitude/phase, STFT axes, and spectral bands all appear in the literature, depending on what constitutes the complementary structure of the signal (Xiao et al., 2024, Zou et al., 2024, Huang et al., 2024). Third, the innovation is often architectural rather than algebraic. TF-Mamba states that its spatial-frequency design is described architecturally rather than through a new custom block equation; the standard Mamba recurrence remains intact, while novelty lies in axis selection, bidirectionality, and residual preservation (Xiao et al., 2024).

A plausible implication is that future progress will depend less on generic multi-direction scanning and more on domain-specific sequence construction, symmetry handling, and fusion rules. Existing papers already point in that direction: DiMSUM uses wavelet ordering and shared attention to counter residual order dependence, SSFMamba exploits conjugate symmetry and phase-aware reconstruction, and Vcamba uses phase-domain motion modeling with semantically coherent frequency scanning (Phung et al., 2024, Zhang et al., 5 Aug 2025, Li et al., 31 Jul 2025). Spatial-Frequency Mamba Blocks, in this sense, are best understood not as a single fixed module, but as a design family that redefines what a Mamba sequence should be before the selective state-space recurrence is applied.