Adaptive Semantic-Frequency SSM

• Adaptive Semantic-Frequency SSM is a state-space module that integrates semantic routing, prompt-pool selection, and frequency-domain context to guide image reconstruction.
• The module modifies only the output projection of Mamba’s state-space equations, preserving transition dynamics while embedding fused semantic-frequency prompts.
• Empirical results show improvements in PSNR and SSIM, indicating enhanced long-range structural and spectral fidelity in infrared image super-resolution tasks.

Searching arXiv for the specified papers to ground the article in published metadata and ensure citation fidelity. Adaptive Semantic-Frequency State Space Module (ASF-SSM) is a state-space augmentation introduced within GPSMamba, a framework for infrared image super-resolution (IRSR), to mitigate limitations associated with the 1D causal scanning mechanism of Mamba on 2D image data. In GPSMamba, ASF-SSM injects a fused semantic-frequency prompt directly into the Mamba block, combining semantic routing, prompt-pool selection, and frequency-domain global context so that reconstruction is guided by both non-local structure and spectral information (Huang et al., 25 Jul 2025). The module is defined as a modification of the output projection in the state-space recurrence rather than a replacement of the underlying Mamba dynamics: $A(t)$ and $B(t)$ remain as in Mamba, while $C(t)$ is modulated by a fused prompt derived from spatial and frequency cues (Huang et al., 25 Jul 2025).

1. Position within GPSMamba

GPSMamba addresses IRSR under the observation that infrared imagery exhibits low contrast and sparse textures, while long-range modeling is required to preserve global coherence. The framework identifies a specific limitation in state-space models such as Mamba: their inherent 1D causal scanning mechanism fragments the global context of 2D images, which hinders fine-detail restoration (Huang et al., 25 Jul 2025). ASF-SSM is proposed as one of two principal components in response to this issue, the other being Thermal-Spectral Attention and Phase Consistency Loss.

Within the model hierarchy, each ASF-SSM replaces the vanilla Mamba SSM scan inside an Adaptive Semantic-Frequency State-Space Block (ASF-SSB). In this setting, ASF-SSM operates on semantic tokens and injects prompt information into the residual branch of the block. The stated objective is to integrate non-local context into the Mamba scan so that reconstruction is not driven solely by a fixed causal order, but by semantic and spectral guidance (Huang et al., 25 Jul 2025).

This design places ASF-SSM at the intersection of prompt-based conditioning and selective state-space modeling. A plausible implication is that the module should be interpreted less as an external side branch and more as an internal control mechanism over the scan output, because its only formal intervention is the modification of the $C(t)$ term.

2. Architectural composition

The high-level block structure begins with an input semantic representation $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$. This tensor is processed by a semantic encoder consisting of an MLP and a Linear layer to produce $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$, which is then split into $\Delta$, $B_{\text{raw}}$, and $C_{\text{raw}}$, each in $\mathbb{R}^{B \times N \times C}$ (Huang et al., 25 Jul 2025). In parallel, the same semantic representation supports two prompt-generation pathways: a Prompt-Pool Router for spatial prompt selection and a Frequency Extractor & Attention branch for global prompt construction.

The Prompt-Pool Router applies Gumbel-Softmax to a linear transform of $B(t)$0, yielding a routing matrix $B(t)$1. This matrix selects prompt vectors from a learnable prompt dictionary $B(t)$2 to form $B(t)$3 (Huang et al., 25 Jul 2025). The frequency branch reshapes $B(t)$4 into 2D patches, applies a 2D FFT, and then uses linear $B(t)$5, $B(t)$6, and $B(t)$7 projections to compute an attention-based global prompt $B(t)$8 (Huang et al., 25 Jul 2025).

The fused prompt is defined as

$B(t)$9

This prompt is then injected into the state-space scan through the modified output projection

$C(t)$0

The recurrence itself is

$C(t)$1

After the scan, the output $C(t)$2 is passed through LayerNorm and Linear, then reshaped back to 2D (Huang et al., 25 Jul 2025).

The architecture therefore combines three operations: semantic encoding for parameter generation, prompt routing for token-conditioned spatial guidance, and spectral attention for non-local global guidance. This suggests that the module is designed to reconcile token-wise selectivity with image-wide context without altering the basic recurrence form of Mamba.

3. Mathematical formulation

The semantic routing stage is specified by

$C(t)$3

followed by

$C(t)$4

The routing matrix selects prompts from the learnable dictionary:

$C(t)$5

where $C(t)$6 (Huang et al., 25 Jul 2025).

The frequency prompt is constructed by first transforming semantic tokens into the spectral domain:

$C(t)$7

Linear projections then form $C(t)$8, $C(t)$9, and $C(t)$0, and the global prompt is obtained through attention:

$C(t)$1

Prompt fusion is additive:

$C(t)$2

The state-space parameters are derived from the split of $C(t)$3:

$C(t)$4

with

$C(t)$5

The output projection is modified as

$C(t)$6

leading to

$C(t)$7

These equations make explicit that ASF-SSM leaves the transition and input terms structurally aligned with Mamba while introducing prompt-conditioned modulation at the readout stage (Huang et al., 25 Jul 2025).

The paper also provides pseudocode-level semantics for fusion: for each token, the spatial prompt is selected by the argmax of the routing distribution, while the global prompt is the attention-weighted aggregation over all tokens. This formalism clarifies that the spatial and spectral components are not alternative pathways but concurrent contributors to a single prompt tensor (Huang et al., 25 Jul 2025).

4. Semantic routing and frequency guidance

A distinctive property of ASF-SSM is that the scan order is no longer the fixed row-major causal order. Tokens are instead processed in a “semantic order” sorted by the routing matrix $C(t)$8 (Huang et al., 25 Jul 2025). This is important because GPSMamba frames causal row-major traversal as a source of fragmented global context in 2D restoration. The reordered scan, coupled with prompt modulation, is intended to reorganize the effective dependencies encountered during recurrence.

The semantic branch is built around a semantic encoder described as an MLP with 2 layers and hidden size $C(t)$9, followed by a Linear projection to $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$0 channels for $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$1, $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$2, and $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$3 (Huang et al., 25 Jul 2025). The routing matrix $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$4 has shape $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$5, where $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$6 is the prompt pool size. The details state “$$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$7,” while the implementation configuration later fixes the prompt pool size at 16 (Huang et al., 25 Jul 2025).

The frequency branch reshapes the token sequence $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$8 with $$X_{\text{semantic}} \in \mathbb{R}^{B \times N \times C}$$9 into 2D patches of size $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$0, applies 2D FFT to obtain $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$1, and then flattens back to a sequence. The $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$2, $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$3, and $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$4 projections each have dimension $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$5, stated as commonly $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$6 and concretely set to 64 in the implementation details (Huang et al., 25 Jul 2025).

This combination of routing and spectral attention establishes the module’s two guiding priors. The spatial prompt localizes token-conditioned semantic categories through a discrete prompt pool, while the global prompt aggregates frequency-domain interactions across the entire image. A plausible implication is that ASF-SSM attempts to bridge local token specialization and global spectral coherence without relying on a purely non-causal self-attention backbone.

5. Training objective and implementation profile

ASF-SSM is trained jointly with the broader GPSMamba framework under a full objective that includes a pixel-wise $$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$7 term and the TSAPC loss. The phase consistency component is defined as

$$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$8

The Thermal-Spectral Attention Loss is given by

$$X_{\text{in}} \in \mathbb{R}^{B \times N \times 3C}$$9

These are combined as

$\Delta$0

and the overall training objective is

$\Delta$1

The details explicitly state that all gradients backpropagate through the ASF-SSM prompt injection (Huang et al., 25 Jul 2025).

The implementation profile reported for GPSMamba specifies batch size 32, number of ASF-SSBs 8, prompt pool size $\Delta$2, feature/channel dimension $\Delta$3, attention head dimension $\Delta$4, and MLP hidden dimension $\Delta$5. Optimization uses Adam with $\Delta$6 and $\Delta$7, an initial learning rate of $\Delta$8 with no decay schedule, Kaiming-normal initialization for all Linear/Conv layers, and FFT zero-padded to match patch grid size (Huang et al., 25 Jul 2025).

These details frame ASF-SSM as a relatively lightweight modification in parameterization terms. That interpretation is reinforced by the paper’s characterization of the module as “lightweight yet powerful,” though the precise computational overhead relative to vanilla Mamba is not quantified in the supplied data (Huang et al., 25 Jul 2025).

6. Empirical behavior and reported effects

The reported ablation on ASF-SSM provides PSNR and SSIM on result-A and result-C at $\Delta$9 scale. The baseline “Mamba (no ASF-SSM, with $B_{\text{raw}}$0 only)” yields 39.03 dB / 39.91 dB and 0.9417 / 0.9528. Adding “+ ASF-SSM (same losses)” yields 39.15 dB / 40.07 dB and 0.9429 / 0.9547. Adding “+ ASF-SSM & TSAPC Loss” yields 39.31 dB / 40.24 dB and 0.9440 / 0.9548 (Huang et al., 25 Jul 2025).

Configuration	PSNR on result-A / result-C	SSIM on result-A / result-C
Mamba (no ASF-SSM, with $B_{\text{raw}}$1 only)	39.03 dB / 39.91 dB	0.9417 / 0.9528
+ ASF-SSM (same losses)	39.15 dB / 40.07 dB	0.9429 / 0.9547
+ ASF-SSM & TSAPC Loss	39.31 dB / 40.24 dB	0.9440 / 0.9548

The details summarize these results as a consistent $B_{\text{raw}}$2–$B_{\text{raw}}$3 dB PSNR gain over the pure Mamba baseline when ASF-SSM is used, and indicate that coupling it with TSAPC Loss drives the overall state-of-the-art performance of GPSMamba (Huang et al., 25 Jul 2025). The same section reports an Effective Receptive Field (ERF) expansion: without ASF-SSM, the ERF is tightly concentrated around each pixel; with ASF-SSM, it becomes anisotropic and far-reaching, which is interpreted as evidence of better long-range fusion (Huang et al., 25 Jul 2025).

Frequency-domain visualizations further associate the loss design with spectral fidelity. Removing TSAPC Loss yields phase-spectrum artifacts described as ghost edges, whereas the full model reproduces a phase spectrum nearly identical to ground truth (Huang et al., 25 Jul 2025). In addition, at $B_{\text{raw}}$4 SR, GPSMamba reconstructs 63% of pixels with error below 5 gray-levels, which is reported as 20% better than the best competing method, while also greatly reducing large errors above 10 (Huang et al., 25 Jul 2025).

These results do not isolate every internal design choice within ASF-SSM, but they do support the paper’s central claim that prompt-guided modulation improves long-range modeling and spectral consistency in IRSR. A plausible implication is that the module’s contribution is better understood through changes in receptive-field structure and error distribution, not only through modest aggregate PSNR increments.

7. Relation to adjacent state-space and frequency-domain designs

ASF-SSM belongs to a broader line of work that combines state-space modeling with explicit frequency-domain processing, but its mechanism differs from approaches that simply prepend or append spectral modules. For example, Wavelet-Mamba for infrared and visible image fusion integrates wavelet transform with an SSM through a Wavelet-SSM module that incorporates wavelet-based frequency domain feature extraction and global information extraction through SSM, while also proposing a cross-modal feature attention modulation (Zhang et al., 24 Mar 2025). That work is motivated by the concern that current methods do not fully combine frequency domain features with global semantic information, resulting in suboptimal extraction of global features across modalities and insufficient preservation of local texture details (Zhang et al., 24 Mar 2025).

The conceptual connection is that both systems treat frequency-domain structure and state-space dynamics as complementary rather than competing components. The difference, based on the available descriptions, is that Wavelet-Mamba emphasizes wavelet-based feature extraction and cross-modal fusion for infrared-visible image fusion, whereas ASF-SSM modifies the internal readout pathway of a Mamba scan by injecting a fused semantic-frequency prompt for super-resolution (Zhang et al., 24 Mar 2025, Huang et al., 25 Jul 2025).

A common misconception would be to regard ASF-SSM as a wholesale replacement of Mamba’s state-space equations. The supplied formulation contradicts that reading: “This is the only change to the original Mamba state-space equations; $B_{\text{raw}}$5 and $B_{\text{raw}}$6 remain as in Mamba, and only $B_{\text{raw}}$7 is modulated by $B_{\text{raw}}$8” (Huang et al., 25 Jul 2025). Another possible misunderstanding is to equate the frequency component with direct supervision alone. In fact, GPSMamba separates the architectural guidance of ASF-SSM from the non-causal supervision of TSAPC Loss, and explicitly presents the combination of the two as a systematic strategy to mitigate the limitations of causal modeling (Huang et al., 25 Jul 2025).

In this sense, ASF-SSM is best understood as an architectural mechanism for prompt-conditioned state-space readout, embedded in a training regime that simultaneously enforces phase and thermal-spectral fidelity.