THAT: Token-wise High-frequency Augmentation Transformer

• The paper’s main contribution is integrating PTSA and MVFN to mitigate attention dispersion and enhance high-frequency details in hyperspectral pansharpening.
• THAT employs token selection and local-window self-attention to efficiently fuse low-res spectral data with high-res panchromatic guidance driven by spectral–spatial priors.
• Empirical results on datasets like Botswana and Pavia show state-of-the-art PSNR, SSIM, and ERGAS scores, validating the approach’s capability in preserving material boundaries and textures.

Token-wise High-frequency Augmentation Transformer (THAT) is a transformer architecture for hyperspectral pansharpening that reconstructs a high-resolution hyperspectral image from a low-resolution hyperspectral observation and a high-resolution panchromatic image by combining token selection with explicit high-frequency augmentation. Introduced in "THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening" (Jin et al., 11 Aug 2025), the method is motivated by two recurrent failure modes of vanilla Vision Transformers in spectral–spatial fusion: attention dispersion over redundant tokens and attenuation of high-frequency structure such as material boundaries, edges, and texture transitions. Its core contribution is the coupling of Pivotal Token Selective Attention (PTSA), which suppresses redundant keys before softmax, with a Multi-level Variance-aware Feed-forward Network (MVFN), which reinforces high-frequency responses across multiple spatial scales.

1. Problem formulation and spectral–spatial priors

Hyperspectral pansharpening seeks to recover a high-resolution hyperspectral image $Y_{hr} \in \mathbb{R}^{sH \times sW \times C}$ from a low-resolution hyperspectral image $X_{lr} \in \mathbb{R}^{H \times W \times C}$ and a high-resolution panchromatic image $P_{hr} \in \mathbb{R}^{sH \times sW}$. A standard forward model writes the low-resolution observation as a spatially degraded version of the target, $X_{lr} \approx D(Y_{hr})$, where $D$ is a blur–downsample operator, and the panchromatic image as a spectral response, $P_{hr} \approx R Y_{hr}$, equivalently $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$, with $R$ encoding the PAN sensor response (Jin et al., 11 Aug 2025).

The training protocol reported for THAT follows Wald’s protocol. In the described setup, $X_{lr}$ is generated by Gaussian blurring with kernels of size $20 \times 20$ or $X_{lr} \in \mathbb{R}^{H \times W \times C}$0 and downsampling by $X_{lr} \in \mathbb{R}^{H \times W \times C}$1, $X_{lr} \in \mathbb{R}^{H \times W \times C}$2, or $X_{lr} \in \mathbb{R}^{H \times W \times C}$3, while $X_{lr} \in \mathbb{R}^{H \times W \times C}$4 is formed by averaging visible bands of the HR-HSI. The datasets are Botswana (Hyperion, 145 cleaned bands), Pavia Centre (ROSIS, 102 bands), and Pavia University (ROSIS, 102 bands).

The model is explicitly motivated by spectral and spatial priors of hyperspectral data. On the spectral side, the paper emphasizes abundance sparsity and strong inter-band correlation, implying substantial spectral redundancy and the existence of a comparatively small set of pivotal tokens carrying informative material signatures. On the spatial side, it emphasizes non-local similarity together with high-frequency structures, which are essential to preserve during pansharpening. This framing is central to THAT: token selection is used to suppress redundancy, while frequency-aware augmentation is used to prevent the smoothing of spatial detail.

2. Network organization and reconstruction pathway

THAT adopts a three-stage architecture composed of shallow feature extraction, deep feature extraction, and reconstruction (Jin et al., 11 Aug 2025). In the shallow stage, the bicubic-upsampled LR-HSI $X_{lr} \in \mathbb{R}^{H \times W \times C}$5 and the HR-PCI $X_{lr} \in \mathbb{R}^{H \times W \times C}$6 are processed by a convolution followed by ReLU, and the shallow features are concatenated with the HR-PCI so that high-resolution spatial guidance is injected at the beginning of the network.

Deep feature extraction is performed by stacking Pivotal Token Selective Groups (PTSGs). Each PTSG contains three components: PTSA, a Window-based Self-Attention (WSA) module, and MVFN. WSA constrains token interactions to local windows, improving locality and computational efficiency, while PTSA and MVFN respectively address redundancy and high-frequency loss. The reconstruction head consists of a convolution followed by pixel-shuffle, producing the fused high-resolution hyperspectral output $X_{lr} \in \mathbb{R}^{H \times W \times C}$7.

The reported implementation uses channel width 180 and 6 attention heads in WSA. Upscaling factors of $X_{lr} \in \mathbb{R}^{H \times W \times C}$8, $X_{lr} \in \mathbb{R}^{H \times W \times C}$9, and $P_{hr} \in \mathbb{R}^{sH \times sW}$0 are considered. The paper states that PTSG is stacked $P_{hr} \in \mathbb{R}^{sH \times sW}$1 times, but $P_{hr} \in \mathbb{R}^{sH \times sW}$2 is not explicitly specified. Likewise, patch size and stride for WSA are not explicitly stated; what is specified is that attention is window-based, so token interactions are localized before PTSA further prunes them.

A useful way to interpret the architecture is that THAT combines three complementary inductive biases. HR-PCI fusion injects high-resolution spatial structure; WSA restricts attention to local neighborhoods where many fine details are formed; and PTSA plus MVFN determine which tokens and which local frequency responses should dominate the subsequent representation.

3. Pivotal Token Selective Attention

The paper characterizes standard transformer attention as vulnerable to attention dispersion in hyperspectral settings. Given $P_{hr} \in \mathbb{R}^{sH \times sW}$3 tokens with embedding dimension $P_{hr} \in \mathbb{R}^{sH \times sW}$4, vanilla attention computes

$P_{hr} \in \mathbb{R}^{sH \times sW}$5

and dense global aggregation can distribute probability mass across many uninformative keys. In hyperspectral images, where smooth regions and inter-band correlation create large numbers of redundant tokens, this dilutes rare but important high-frequency cues.

PTSA addresses this by constructing a query-specific binary mask over keys before softmax (Jin et al., 11 Aug 2025). Raw attention logits are formed as

$P_{hr} \in \mathbb{R}^{sH \times sW}$6

where $P_{hr} \in \mathbb{R}^{sH \times sW}$7 is learnable. For stability, the features are normalized row-wise, $P_{hr} \in \mathbb{R}^{sH \times sW}$8 and $P_{hr} \in \mathbb{R}^{sH \times sW}$9, and in practice $X_{lr} \approx D(Y_{hr})$0 can be computed from $X_{lr} \approx D(Y_{hr})$1. For each query token $X_{lr} \approx D(Y_{hr})$2, PTSA applies k-means with $X_{lr} \approx D(Y_{hr})$3 to the similarity scores $X_{lr} \approx D(Y_{hr})$4. The cluster with the larger mean similarity is labeled pivotal, yielding a binary mask $X_{lr} \approx D(Y_{hr})$5.

The masked logits are then

$X_{lr} \approx D(Y_{hr})$6

followed by masked softmax,

$X_{lr} \approx D(Y_{hr})$7

and the attention output

$X_{lr} \approx D(Y_{hr})$8

In multi-head form, each head performs its own row-wise clustering and masking, after which the head outputs are concatenated and projected in the standard way.

The stated objective of PTSA is not merely sparsity but de-dispersion. The paper’s complexity discussion notes that vanilla attention requires $X_{lr} \approx D(Y_{hr})$9 to form $D$0 and another $D$1 for $D$2, whereas PTSA adds row-wise k-means with $D$3 and small iteration count $D$4, approximately $D$5, but reduces the post-mask aggregation cost to $D$6 and the softmax cost to $D$7, where $D$8 is the number of pivotal keys per query. In the reported interpretation, the primary gain is better quality through attention concentration, with computational savings becoming more visible when combined with local windows.

4. Multi-level variance-aware augmentation

MVFN replaces the standard feed-forward network with a multi-branch module tailored to high-frequency enhancement (Jin et al., 11 Aug 2025). Its specified components are three depthwise convolutions with kernel sizes 3, 5, and 7; variance modeling modules denoted Var-3, Var-5, and Var-7; pooling on each branch; concatenation; SiLU activation; a final convolution; and a residual connection. The stated purpose is to emphasize edges and textures over flat, redundant responses.

The paper does not present closed-form equations for MVFN, but its operations can be written for clarity in a variance-aware FFN form. Let $D$9 denote window tokens reshaped to spatial maps. For each kernel size $P_{hr} \approx R Y_{hr}$0, a branch computes a spatial feature

$P_{hr} \approx R Y_{hr}$1

a local variance map

$P_{hr} \approx R Y_{hr}$2

a gate

$P_{hr} \approx R Y_{hr}$3

and a gated branch output

$P_{hr} \approx R Y_{hr}$4

The branches are aggregated as

$P_{hr} \approx R Y_{hr}$5

where $P_{hr} \approx R Y_{hr}$6 is SiLU.

Within THAT, high-frequency augmentation is therefore distributed across two mechanisms. PTSA increases the probability that informative tokens dominate attention, especially near edges and material transitions. MVFN then augments those representations through multi-scale depthwise convolutions modulated by local variance estimates, which act as branch-wise detectors of high-frequency energy. The architecture thereby treats token selection and frequency enhancement as coupled operations rather than independent modules.

5. Training protocol and empirical performance

The reported implementation uses PyTorch on a single NVIDIA RTX 3090 with 24 GB memory, Adam with weight decay 0, an initial learning rate of $P_{hr} \approx R Y_{hr}$7 halved every 20 epochs, 50 epochs of training, batch size 2, and a single $P_{hr} \approx R Y_{hr}$8 supervision loss,

$P_{hr} \approx R Y_{hr}$9

No additional spectral, spatial, adversarial, or total-variation losses are used (Jin et al., 11 Aug 2025).

Evaluation is performed on Botswana, PaviaC, and PaviaU at $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$0, $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$1, and $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$2, using PSNR, SSIM, SAM, ERGAS, and SCC. At $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$3, THAT reports 29.18 PSNR, 0.9084 SSIM, 2.6657 SAM, 2.0754 ERGAS, and 0.9493 SCC on Botswana; 35.29 PSNR, 0.9574 SSIM, 5.2818 SAM, 1.8838 ERGAS, and 0.9792 SCC on PaviaC; and 37.82 PSNR, 0.9632 SSIM, 3.0172 SAM, 1.0039 ERGAS, and 0.9816 SCC on PaviaU. The paper states that on PaviaC these are the best PSNR, SSIM, ERGAS, and SCC, and on PaviaU the best PSNR, SSIM, and ERGAS. For Botswana at $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$4, the PSNR is comparable to PSDip at 29.20, while SSIM, SAM, and SCC are markedly improved.

At $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$5, THAT reports 29.34 PSNR, 0.8728 SSIM, 3.8377 SAM, 2.3373 ERGAS, and 0.8979 SCC on Botswana; 32.43 PSNR, 0.9157 SSIM, 7.4978 SAM, 2.5841 ERGAS, and 0.9420 SCC on PaviaC; and 32.69 PSNR, 0.9102 SSIM, 5.0876 SAM, 1.7676 ERGAS, and 0.9175 SCC on PaviaU. The paper describes the Botswana result as best or second-best across metrics and states that it outperforms TTST in SSIM by $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$6 and SCC by $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$7. On PaviaC, the reported numbers are the best PSNR, SSIM, ERGAS, and SCC, and on PaviaU the best PSNR and ERGAS.

At $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$8, THAT reports 30.92 PSNR, 0.8918 SSIM, 4.0053 SAM, 1.9397 ERGAS, and 0.9147 SCC on Botswana; 29.22 PSNR, 0.8499 SSIM, 11.7926 SAM, 3.6820 ERGAS, and 0.8178 SCC on PaviaC; and 31.61 PSNR, 0.8982 SSIM, 5.1381 SAM, 2.0157 ERGAS, and 0.8973 SCC on PaviaU. The paper states that the Botswana $$P_{hr}(x) \approx \sum_{c=1}^{C} r_c \cdot Y_{hr}^{(c)}(x)$$9 result is best across all metrics against TTST, whose PSNR is 30.73; that PaviaC achieves the best PSNR and ERGAS against TTST at 28.14 PSNR; and that PaviaU attains the best PSNR and ERGAS. Band-wise PSNR plots on Botswana, PaviaC, and PaviaU at $R$0 are described as showing THAT leading on most bands, which the paper interprets as evidence of robust spectral fidelity across wavelengths.

6. Ablations, efficiency, related frequency-aware token methods, and limitations

The ablation evidence attributes performance to the joint contribution of PAN guidance, PTSA, and MVFN rather than to any single submodule (Jin et al., 11 Aug 2025). Removing HR-PCI causes large drops: at $R$1 on PaviaU, PSNR falls from 37.82 to 33.24 and ERGAS rises from 1.0039 to 1.6454; at $R$2 on Botswana, PSNR falls from 30.92 to 24.77. Removing PTSA also produces consistent degradation: on PaviaC $R$3, PSNR drops from 29.22 to 28.34 and ERGAS rises from 3.6820 to 4.0366; on Botswana $R$4, PSNR drops from 30.92 to 30.28 and ERGAS rises from 1.9397 to 2.2157. Removing MVFN produces further losses associated with high-frequency reconstruction: on Botswana $R$5, PSNR drops from 29.18 to 28.24 and ERGAS rises from 2.0754 to 2.6411; on PaviaC $R$6, PSNR drops from 35.29 to 34.13 and ERGAS rises from 1.8838 to 2.1607. A separate comparison replacing MVFN with MFL also degrades performance, for example from 37.82/0.9632/1.0039 to 36.99/0.9585/1.1018 on PaviaU $R$7 and from 30.92/0.8918/1.9397 to 29.91/0.8651/2.2035 on Botswana $R$8.

In parameter and FLOP terms, the reported Botswana comparison is as follows:

Model	Parameters	FLOPs
THAT	1.45 M	78.42 G
TTST	1.32 M	92.17 G
DHP-DARN	0.47 M	30.38 G
DBDENet	1.32 M	117.76 G
PLRDiff	391.05 M	22.43 T

The paper positions THAT against several earlier transformer baselines. It argues that TTST’s iterative top-k search is ratio-sensitive and costly, that SSA sparsifies logits without explicit spatial-frequency modeling, and that FW-SAT introduces pointer overheads. THAT’s novelty is located in the combination of a clustering-based token selector, local-window attention, and a variance-aware multi-scale FFN.

A common simplification is to assimilate THAT to later frequency-aware token pruning frameworks. That is not the formulation reported in the pansharpening paper. "Frequency-Aware Token Reduction for Efficient Vision Transformer" (Lee et al., 26 Nov 2025) partitions tokens into high-frequency and low-frequency sets using an attention-derived high-pass score, preserves the former, and aggregates the latter into DC tokens in generic ViT backbones. "HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing" (You et al., 11 Jun 2026) uses mask-aware high-frequency token selection plus a low-frequency branch to accelerate DiT-based image editing. "Spectral Image Tokenizer" (Esteves et al., 2024) instead tokenizes images in a coarse-to-fine wavelet spectrum. These works share a frequency-aware perspective, but THAT, as introduced for hyperspectral pansharpening, is distinguished by PTSA’s row-wise k-means masking and MVFN’s variance-aware augmentation inside a PAN-guided reconstruction network. This suggests a broader family of token methods organized around explicit treatment of high-frequency information, but the mechanisms and target tasks differ substantially.

The limitations reported for THAT are correspondingly specific. PTSA uses per-query k-means with $R$9, and the paper notes possible sensitivity to clustering stability in very flat regions or under extremely noisy spectra. The selection stage adds overhead of order $X_{lr}$0 per head per window beyond $X_{lr}$1, even if iteration counts are small. Generalization across sensors may be affected because the PAN signal in the experiments is constructed by averaging visible bands rather than using a real sensor-specific spectral response $X_{lr}$2. Finally, although WSA and PTSA alleviate local quadratic costs, very large scenes still present memory pressure; the paper points to hierarchical tiling, streaming inference, and low-rank attention as future directions.