H-S³A: Hierarchical Spectral-Spatial Attention

• H-S³A is a neural attention mechanism engineered for hyperspectral imaging that enforces both spectral fidelity and spatial detail through hierarchical processing.
• It employs structured spectral grouping, trilateral/multi-branch attention, and boundary channel shuffling to effectively capture cross-band and spatial dependencies.
• Its plug-and-play design integrates with various backbones, significantly boosting performance in hyperspectral super-resolution and multi-modal classification tasks.

Hierarchical Spectral-Spatial Synergy Attention (H-S³A) is a class of neural attention mechanisms purpose-built for hyperspectral image (HSI) modeling, targeting the unified reinforcement of spatial detail and spectral fidelity. Its principal contributions are the structured, multi-level processing of spectral groups; explicit modeling of cross-band and spatial correlations; and architectural flexibility enabling seamless integration into a variety of backbone networks. Two distinct but structurally analogous realizations have been proposed: one as a plugin in the SR²-Net pipeline for hyperspectral super-resolution (He et al., 29 Jan 2026), and another, termed the “Hierarchical Attention Module” (HAM), within HAPNet for HSI+SAR multi-source data classification (Luo et al., 2024).

1. Motivation and Fundamental Objectives

HSI processing mandates the recovery of spatial structure (edges, textures) while enforcing spectral consistency—preservation of physically plausible, smooth, and artifact-free spectra across spatial locations. Classical RGB backbones and standard attention primitives inadequately address inter-band dependencies, treating the spectral axis as a mere stack of independent channels, frequently inducing cross-band artifacts and spectral misalignments. H-S³A is designed to inject deep cross-band interaction—jointly leveraging spectral context and spatial granularity—prior to any further manifold-based spectral rectification or cross-modal fusion, thereby improving both data fidelity and cross-domain transferability (He et al., 29 Jan 2026, Luo et al., 2024).

2. Architectural Blueprint and Workflow

The H-S³A block is hierarchically stacked (typically $B=4$ layers (He et al., 29 Jan 2026) or $L=3$ layers (Luo et al., 2024)) and follows a modular sequence:

1. Spectral Grouping: The input is partitioned into $G$ contiguous spectral groups ($G=4$ by default (He et al., 29 Jan 2026)) or processed in full-channel mode (HAM (Luo et al., 2024)), enabling local spectral context modeling.
2. Trilateral/Multigranular Attention: Each group (or full channel stack) is processed by a dedicated attention unit. In SR²-Net, a Trilateral Synergy Attention (TSA) mechanism is used to capture spatial ($H,W$) and spectral ($S$) interdependencies via three summary attention maps; in HAPNet, the block is decomposed into global (spatial), spectral, and local branches, each employing self-attention or depthwise convolutions.
3. Boundary Channel Shuffling: To ensure information mixing across adjacent spectral groups, group boundaries are shuffled (by swapping interface bands), thus mitigating discontinuities and further smoothing spectral responses (He et al., 29 Jan 2026).
4. $1\times1$ Convolutional Fusion: The outputs are fused back to the original channel dimensionality, allowing the next H-S³A block to receive a full-spectrum, synergy-enhanced feature map.
5. Inter-Block Fusion and Downstream Rectification: The final features are passed—either into a manifold consistency rectifier (SR²-Net) or a frequency-domain parallel fusion unit for multi-source data (HAPNet)—to further enhance spectral consistency or modality alignment.

3. Mathematical Formulation of Attention Operations

SR²-Net (Trilateral Synergy Attention, TSA) (He et al., 29 Jan 2026)

For group feature $G'\in \mathbb{R}^{H\times W\times S/G}$:

• Compute average-pooled projections along each axis ($d\in\{h,w,s\}$):

$$A^d = \sigma\left(\mathrm{Conv}_2\left(\mathrm{GeLU}\left(\mathrm{Conv}_1(\mathrm{AvgPool}_d(G'))\right)\right)\right)$$

where $A^h\in\mathbb{R}^{H\times1\times S/G}$, $A^w\in\mathbb{R}^{1\times W\times S/G}$, $A^s\in\mathbb{R}^{1\times 1\times S/G}$.

• Fuse attention:

$$F = G' \odot \left(\alpha_h A^h + \alpha_w A^w + \alpha_s A^s\right)$$

with $\alpha_h,\alpha_w,\alpha_s$ as learnable scalars.

HAPNet (HAM) (Luo et al., 2024)

For input $X\in\mathbb{R}^{B\times C\times H\times W}$ (flattened as needed):

• Global Branch: Anchored self-attention over $H\times W$ spatial plane using averaged anchor tokens and softmax dot-product weighting.
• Spectral Branch: Anchored self-attention over $C$, the channel/spectral dimension.
• Local Branch: Depthwise convolutions (kernel $3\times3$) with channel attention gates (squeeze-and-excitation).
• Fused output is elementwise summed and forwarded through a two-layer FFN with GELU activation and finalized by LayerNorm.

4. Integration into Broader Networks

Enhance-Then-Rectify Flow (SR²-Net) (He et al., 29 Jan 2026)

The H-S³A module is strategically positioned to process backbone (e.g. SwinIR) outputs before physically constrained rectification. The pipeline is:

$$I_{\rm LR} \xrightarrow{f_{\rm SR}} \tilde I_{\rm SR} \xrightarrow{\mathrm{H\!-\!S}^3\mathrm A} F_s \xrightarrow{\rm MCR} \hat I_{\rm SR}$$

• H-S³A delivers spectrally consistent, detail-rich intermediate features $F_s$.
• MCR projects $F_s$ to a low-dimensional spectral manifold and iteratively refines the spectra, ensuring physical plausibility.

Multi-Source Classification (HAPNet) (Luo et al., 2024)

• Stacked H-S³A modules extract multi-granularity HSI features.
• These features are fused with SAR representations using a Parallel Filter Fusion Module (PFFM); fusion occurs in both spatial and frequency domains, passing through $2D$-FFT modules and learnable global frequency filters.
• Final concatenated outputs are classified through fully connected layers.

5. Hyper-Parameters, Ablations, and Performance Metrics

Key Hyper-Parameters

Parameter	SR²-Net (He et al., 29 Jan 2026)	HAPNet (Luo et al., 2024)
Spectral groups $G$	4	— (full-channel)
H-S³A blocks $B$	4	3
MCR stages $N$	1	—
Manifold rank $r$	8	—

• Convolution kernel sizes in H-S³A blocks in SR²-Net: $\{3,5,7,3\}$ (group-specific, multi-scale extraction).
• Loss weights in SR²-Net: $\lambda_{\rm rec}=1.0$, $\lambda_{\rm deg}=0.2$ (enforces bicubic-downsample consistency).

Ablation Insights

SR²-Net (ARAD-1K, SwinIR backbone, ×4 scale) (He et al., 29 Jan 2026):

• No H-S³A, no MCR: mPSNR 39.5717, mSAM 1.3950
• H-S³A only: mPSNR 40.7059 (+1.13 dB), mSAM 1.3476
• MCR only: mPSNR 40.2550, mSAM 1.3173
• H-S³A + MCR: mPSNR 40.9720, mSAM 1.2819

HAPNet multi-source classification (Luo et al., 2024):

• Without H-S³A: OA drops from 91.44%→90.35% (−1.09%) on Augsburg, 80.51%→74.49% (−6.02%) on Berlin.
• Without PFFM: OA drops from 91.44%→89.80% (−1.64%) on Augsburg, 80.51%→76.75% (−3.76%) on Berlin.

H-S³A contributes significant performance increases in both super-resolution (up to +2.4 dB mPSNR) and classification (+1.09–6.02% OA).

6. Structural and Computational Characteristics

H-S³A is explicitly lightweight:

• Negligible overhead: +0.05 M parameters, +1.48 GFLOPs when added to SwinIR-×4 (He et al., 29 Jan 2026).
• All convolutions are $1\times1$ or $3\times3$ (depthwise in HAPNet), with per-block parameterization.
• TSA omits normalization layers; only uses $1\times1$ convolutions, GeLU, and sigmoid.
• All attention and fusion scalars are learnable, allowing dynamic re-weighting.

The module is plug-and-play with respect to diverse backbones and does not impose architectural modifications, thus remaining generalizable and portable across tasks involving spectral-spatial reasoning.

7. Comparative Perspective and Significance

H-S³A's core distinction lies in its explicit encoding of both local and global dependencies across multiple axes—spatial semantics and spectral continuity, and in the hierarchical structuring of this synergy. Compared to prior single-axis attentions or spatially-focused convolutional methods, H-S³A reduces cross-band artifacts and enforces physical spectral plausibility more robustly. Its success across disparate modalities (super-resolution, multimodal classification), and its negligible computational tax, underscore its utility as a general module for spectral–spatial modeling in next-generation hyperspectral image processing networks (He et al., 29 Jan 2026, Luo et al., 2024).