Propagation-Guided Spectral Video Transformer

• The paper introduces PG-SVRT, a novel video-level SCI reconstruction method that leverages propagation-guided mechanisms to recover masked spectral information and improve temporal consistency.
• It employs a U-Net-based architecture centered on Mask-Guided Degradation Perception and Cross-Domain Propagated Blocks to address spatial-spectral loss and frame-by-frame inconsistencies.
• Experimental results on DD-CASSI systems demonstrate superior PSNR, SSIM, and reduced SAM with low computational cost, validating the efficacy of the proposed approach.

Searching arXiv for the specified paper and closely related reference works. Propagation-Guided Spectral Video Reconstruction Transformer (PG-SVRT) is a video-level reconstruction network for Spectral Compressive Imaging (SCI) that estimates a spectral video

$X \in \mathbb{R}^{T \times H \times W \times C}$

from a sequence of compressed measurements $Y \in \mathbb{R}^{T \times H \times W'}$. It is introduced together with DynaSpec, which is described as the first high-quality dataset for dynamic hyperspectral video reconstruction. The method is motivated by two limitations of predominantly image-based SCI reconstruction: masked spatial-spectral information loss caused by coded measurement formation, and temporal inconsistency caused by frame-by-frame recovery. PG-SVRT addresses these issues with a U-Net-based architecture centered on Mask-Guided Degradation Perception (MGDP) and Cross-Domain Propagated Blocks (CDPBs), using spatial-then-temporal attention and a bridged-token mechanism to reduce computational complexity while improving reconstruction quality, spectral fidelity, and temporal consistency (Cai et al., 28 Feb 2026).

1. Problem setting and optical formulation

PG-SVRT is formulated for video-level compressive spectral reconstruction in SCI rather than single-frame hyperspectral recovery. In the paper’s setting, a latent hyperspectral frame is denoted $X_i \in \mathbb{R}^{H \times W \times C}$, while the video-level target is $X \in \mathbb{R}^{T \times H \times W \times C}$. The corresponding compressed measurement sequence is $Y \in \mathbb{R}^{T \times H \times W'}$. The shift from image-level to video-level reconstruction is explicitly presented as more than a batched reformulation: the model must exploit both intra-frame spatial-spectral structure and inter-frame temporal continuity and complementarity (Cai et al., 28 Feb 2026).

The sensing model depends on the SCI optical architecture. For single-disperser (SD) systems such as SD-CASSI and PMVIS, the paper gives

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$

For dual-disperser (DD) systems such as DD-CASSI and NDSSI, it gives

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$

Both are unified as

$Y_i = \Psi X_i + \Theta,$

where $\Psi$ is the encoding operator and $\Theta$ is noise. In SD systems,

$Y \in \mathbb{R}^{T \times H \times W'}$0

whereas in DD systems,

$Y \in \mathbb{R}^{T \times H \times W'}$1

The paper benchmarks four representative SCI systems: PMVIS, SD-CASSI, NDSSI, and DD-CASSI. Their reported differences are operationally important for PG-SVRT because temporal propagation quality depends on the structural cues preserved by the optics. DD-CASSI is reported as best suited for video-level reconstruction under the tested settings because it offers higher spectral sampling efficiency, clearer structural representation, and better support for temporal propagation. By contrast, PMVIS and SD-CASSI are reported to lack enough spatial cues for temporal propagation, and NDSSI is reported to have limited spectral sampling density (Cai et al., 28 Feb 2026).

A central conceptual point is that adjacent measurements under a fixed encoding pattern contain complementary information. The paper’s claim is that neighboring frames may reveal features that help disambiguate masked content that is uncertain in any single compressed observation. This is the immediate rationale for propagation-guided video reconstruction.

2. DynaSpec dataset and benchmark construction

DynaSpec is presented as the first high-quality dataset for dynamic hyperspectral video reconstruction. Its purpose is to provide ground truth for video-level SCI reconstruction, addressing the limitations of image-based SCI datasets and pseudo-video generation from static hyperspectral images. The paper argues that static-image cropping only simulates limited rigid motion such as camera shift and does not capture genuine scene dynamics or articulated motion (Cai et al., 28 Feb 2026).

The dataset is acquired with a GaiaField push-broom hyperspectral camera. Because true snapshot ground-truth spectral video is difficult to obtain, the paper uses frame-scanning acquisition: each hyperspectral frame is captured individually, and dynamic scenes are emulated by manually introducing object motion between frames. The designed motions include translation, rotation, and articulated movement.

The reported dataset statistics are:

• 30 scenes
• 300 HSIs total
• spatial resolution $Y \in \mathbb{R}^{T \times H \times W'}$2
• spectral resolution 2 nm
• wavelength range 400–700 nm
• 151 spectral channels

For the reconstruction experiments, the evaluated bands are matched to the real prototype and use 30 spectral channels from 500 nm to 650 nm.

The simulation protocol uses CAVE and 25 DynaSpec sequences for training, and KAIST together with the remaining DynaSpec samples for testing. No explicit validation split is stated. Cropping follows a previously used SCI strategy: training crops are $Y \in \mathbb{R}^{T \times H \times W'}$3, CAVE is cropped using random step sizes, and DynaSpec uses a 70% probability of step size 0 because it already contains real dynamic content. Test crops on KAIST and DynaSpec use pre-generated random step sizes.

The paper also states five acquisition principles intended to ensure reliability:

1. Motion between consecutive frames is continuous and physically plausible.
2. Long integration times are used to mitigate noise.
3. Spectral correction is applied based on the camera spectral response.
4. Illumination spectral properties are excluded so the data approximates reflectance.
5. Intensity calibration uses invariant objects in the sequence to reduce drift, especially temperature drift.

A limitation explicitly acknowledged in the paper is that DynaSpec is a controlled benchmark with indoor lighting and specific motion statistics, which may limit generalization to unscripted natural environments. The paper does not describe an explicit post-hoc geometric registration algorithm in the main text; alignment is described as being handled mainly by acquisition design and calibration (Cai et al., 28 Feb 2026).

3. Network organization and the meaning of “propagation-guided”

PG-SVRT is a U-Net-based architecture composed primarily of Mask-Guided Degradation Perception (MGDP) and Cross-Domain Propagated Blocks (CDPBs). Each CDPB contains Cross-Domain Propagated Attention (CDPA) and a Multi-Domain Feed-Forward Network (MDFFN). The paper also mentions a shuffle operation to align degradation features with measurements across the spectral dimension, but the exact implementation of the shuffle is not specified in the text (Cai et al., 28 Feb 2026).

The processing pipeline is described as follows:

1. Input a sequence of compressed measurements $Y \in \mathbb{R}^{T \times H \times W'}$4 and the coding mask $Y \in \mathbb{R}^{T \times H \times W'}$5.
2. Use MGDP to model the degradation process and produce degradation-aware features.
3. Concatenate these features with the original measurements to form $Y \in \mathbb{R}^{T \times H \times W'}$6.
4. Feed $Y \in \mathbb{R}^{T \times H \times W'}$7 into a U-Net backbone composed of multiple CDPBs.
5. Within each CDPB, apply spatial attention first, then temporal attention, then MDFFN.
6. Decode the result to output the reconstructed spectral video.

The implementation uses

$Y \in \mathbb{R}^{T \times H \times W'}$8

basic channel size

$Y \in \mathbb{R}^{T \times H \times W'}$9

and frame number

$X_i \in \mathbb{R}^{H \times W \times C}$0

The text does not elaborate further on the exact meaning of $X_i \in \mathbb{R}^{H \times W \times C}$1 beyond indicating module counts in the U-Net hierarchy.

The phrase “Propagation-Guided” has a specific meaning in this architecture. It does not refer to optical-flow-based recurrent hidden-state propagation of the kind used in some video restoration models. Instead, it refers to feature propagation across spatial and temporal domains inside the attention mechanism. The most explicit instance is CDPA, where the output of spatial attention is used as the value input to temporal attention. Spatially reconstructed features are therefore propagated into temporal reasoning, rather than treating spatial and temporal attention as isolated stages.

This point matters because the paper explicitly distinguishes PG-SVRT from classical frame-by-frame or flow-based propagation. The model is intended to exploit the fact that adjacent compressed measurements contain complementary information under a fixed encoding pattern. In that sense, the propagation signal is an attention-mediated cross-domain representation.

4. Core modules: MGDP, CDPA, bridged tokens, and MDFFN

MGDP

MGDP is designed to encode the degradation prior induced by the SCI optics before the main transformer processing. The mask matrix is defined as

$X_i \in \mathbb{R}^{H \times W \times C}$2

For SD systems, each channel uses $X_i \in \mathbb{R}^{H \times W \times C}$3; for DD systems, the effective mask at channel $X_i \in \mathbb{R}^{H \times W \times C}$4 is

$X_i \in \mathbb{R}^{H \times W \times C}$5

MGDP first compresses $X_i \in \mathbb{R}^{H \times W \times C}$6 along the spectral dimension to obtain

$X_i \in \mathbb{R}^{H \times W \times C}$7

It then crops or replicates $X_i \in \mathbb{R}^{H \times W \times C}$8 to form

$X_i \in \mathbb{R}^{H \times W \times C}$9

which represents the spatial intensity distribution of each channel after degradation. A $X \in \mathbb{R}^{T \times H \times W \times C}$0 convolution learns the intensity distribution difference between $X \in \mathbb{R}^{T \times H \times W \times C}$1 and $X \in \mathbb{R}^{T \times H \times W \times C}$2, followed by a sigmoid to compute a weight map. Similar processing is applied to the measurement sequence $X \in \mathbb{R}^{T \times H \times W \times C}$3, and the final input to the main architecture is

$X \in \mathbb{R}^{T \times H \times W \times C}$4

CDPA

CDPA is the architectural core of PG-SVRT. For the first CDPB, given

$X \in \mathbb{R}^{T \times H \times W \times C}$5

the model computes

$X \in \mathbb{R}^{T \times H \times W \times C}$6

with

$X \in \mathbb{R}^{T \times H \times W \times C}$7

The model applies spatial attention first, then temporal attention. The paper states that spatial reconstruction is the first bottleneck because SCI encoding masks and aliases spectral-spatial structure within each frame; temporal attention then propagates the improved representation across frames.

For spatial attention, features are partitioned into non-overlapping windows,

$X \in \mathbb{R}^{T \times H \times W \times C}$8

with intended

$X \in \mathbb{R}^{T \times H \times W \times C}$9

and the chosen rectangular window

$Y \in \mathbb{R}^{T \times H \times W'}$0

Bridged token mechanism

To reduce the cost of spatial attention, PG-SVRT introduces a bridged token

$Y \in \mathbb{R}^{T \times H \times W'}$1

where $Y \in \mathbb{R}^{T \times H \times W'}$2 is the number of bridge tokens. The paper states that $Y \in \mathbb{R}^{T \times H \times W'}$3 essentially represents pooled information from $Y \in \mathbb{R}^{T \times H \times W'}$4, and is generated by directly pooling $Y \in \mathbb{R}^{T \times H \times W'}$5.

The spatial attention is written as

$Y \in \mathbb{R}^{T \times H \times W'}$6

with

$Y \in \mathbb{R}^{T \times H \times W'}$7

where $Y \in \mathbb{R}^{T \times H \times W'}$8 is a learnable temperature parameter.

For temporal processing, the queries and keys are rearranged as

$Y \in \mathbb{R}^{T \times H \times W'}$9

A distinctive design choice is that temporal attention reuses the spatially updated feature as the value rather than computing a new value projection:

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$0

This is the paper’s explicit propagation mechanism: spatially reconstructed features are propagated into temporal attention through shared values.

The paper also states that temporal windowing is not used. The reason given is task-specific: useful frame correlation is already local because $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$1 is small, and temporal windows would require padding and could create signal interference or redundant computation.

Complexity and MDFFN

The complexity of CDPA is given as

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$2

The condition for complexity benefit is

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$3

With

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$4

and

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$5

the condition is satisfied:

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$6

After CDPA, PG-SVRT uses MDFFN rather than a standard transformer FFN. The paper states that MDFFN is inspired by multi-head mechanisms and decomposed 3D CNN ideas, divides spectral features into different heads, performs self-attention in both spatial and temporal domains, and then fuses them. The main text does not provide a full algebraic formulation of MDFFN, but its role is explicitly described as enhancing intra-domain feature extraction while promoting cross-domain fusion (Cai et al., 28 Feb 2026).

5. Training regime, evaluation protocol, and quantitative results

Training is supervised with full hyperspectral targets. The paper uses CAVE static hyperspectral images and 25 DynaSpec sequences for training, and KAIST plus the remaining DynaSpec sequences for testing. All experimental bands are aligned with the real DD-CASSI prototype and use 30 spectral channels over 500 nm to 650 nm. The spatial crop size is $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$7 and the sequence length is $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$8 (Cai et al., 28 Feb 2026).

The implementation details reported are:

• framework: PyTorch
• GPUs: RTX 3090
• epochs: 80
• batch size: 2
• optimizer: Adam
• $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w) \cdot X_i(h, w - \sigma(c), c).$$9, $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$0
• initial learning rate:

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$1

• final learning rate under cosine annealing:

$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$2

The paper states that training uses a multi-stage RMSE loss, but the exact mathematical formula is not provided in the main text.

Evaluation uses PSNR, SSIM, SAM, ST-RRED, Params, and GFLOPs. The paper notes that all metrics except Params are reported as frame-wise averages. Higher PSNR and SSIM are better, while lower SAM and ST-RRED indicate better spectral fidelity and temporal consistency.

In the main benchmark table, PG-SVRT is reported with:

• 2.48M parameters
• 28.18 GFLOPs

The paper compares this with image-based SCI baselines including MST-L, CST-L, DAUHST, GAP-Net, DADF-Plus-3, DPU, and DPU$$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$3, and states that PG-SVRT achieves stronger reconstruction and temporal consistency while remaining efficient.

On KAIST, PG-SVRT reports:

• PSNR: 41.23
• SSIM: 0.9882
• SAM: 3.805
• ST-RRED: 19.35

On DynaSpec, it reports:

• PSNR: 41.82
• SSIM: 0.9904
• SAM: 4.0118
• ST-RRED: 27.14

The paper states that the largest and most meaningful gains are in SAM and ST-RRED, supporting the claim that video-level propagation helps recover masked spectral information and improve temporal smoothness.

Before algorithm comparison, the paper also benchmarks four SCI systems under a unified reconstruction framework using PG-SVRT. The reported results are:

• PMVIS: PSNR 28.45, SSIM 0.8456, SAM 5.4162, ST-RRED 459.49
• SD-CASSI: PSNR 37.78, SSIM 0.9700, SAM 4.0737, ST-RRED 23.21
• NDSSI: PSNR 37.84, SSIM 0.9825, SAM 5.4091, ST-RRED 91.8
• DD-CASSI: PSNR 41.52, SSIM 0.9893, SAM 3.9084, ST-RRED 23.25

From these results, the paper concludes that DD-CASSI is the best sensing architecture for video-level spectral reconstruction under the tested settings.

The real-world benchmark uses a DD-CASSI prototype, five real measurement sequences, and spatial size $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$4. All methods are retrained on DynaSpec using the real mask before evaluation. Qualitatively, the paper reports fewer artifacts, more accurate spectral curves, smaller error maps, better temporal smoothness, and cleaner pseudo-RGB renderings (Cai et al., 28 Feb 2026).

6. Ablations, limitations, and position relative to neighboring methods

The ablation studies identify the main contributors to PG-SVRT’s behavior. Starting from a baseline using spatially windowed F-MSA and a regular FFN, the paper reports:

• baseline: PSNR 39.97, ST-RRED 43.90
• + CDPA: PSNR 41.30, ST-RRED 25.44
• + MGDP: PSNR 41.41, ST-RRED 24.63
• + MDFFN: PSNR 41.52, ST-RRED 23.25

This attributes the largest gain, especially in temporal consistency, to CDPA.

For spatiotemporal ordering, the paper compares parallel, temporal-then-spatial, temporal-then-spatial with propagation, spatial-then-temporal, and spatial-then-temporal with propagation. The best strategy is spatial-then-temporal with propagation, with:

• PSNR 41.52
• SSIM 0.9893
• SAM 3.9084
• ST-RRED 23.25

For bridge-token count, the paper reports:

• None: PSNR 41.18, ST-RRED 27.90, 32.20 GFLOPs
• $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$5: PSNR 41.11, 25.26 GFLOPs
• $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$6: best overall, PSNR 41.52, ST-RRED 23.25, 28.18 GFLOPs
• $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$7: still good but much more expensive, 33.10 GFLOPs

For MDFFN, the reported ablation is:

• regular Conv3d FFN: PSNR 41.41, ST-RRED 24.63
• without temporal branch: PSNR 41.16, ST-RRED 26.71
• without spatial branch: PSNR 41.42, ST-RRED 26.18
• full MDFFN: PSNR 41.52, ST-RRED 23.25

These results support the paper’s claim that both spatial and temporal processing matter, and that their fusion gives the best reconstruction and temporal consistency.

The paper also makes two important delimitations. First, PG-SVRT is not merely an image SCI model with added temporal attention; it is framed as a transition from image-level to video-level reconstruction. Second, its propagation mechanism is not the optical-flow-based recurrent propagation used in classical video restoration. This is a useful contrast with adjacent literature.

Relative to VRT, which is a general video restoration transformer with temporal mutual self attention and parallel warping for tasks such as video super-resolution, deblurring, denoising, frame interpolation, and space-time super-resolution, PG-SVRT targets SCI-specific spectral video reconstruction and introduces mask-guided degradation modeling and cross-domain propagated attention instead of VRT’s alignment-aware but non-spectral video restoration design (Liang et al., 2022). Relative to RVRT, which processes local neighboring frames in parallel within a globally recurrent framework and uses guided deformable attention for clip-to-clip alignment, PG-SVRT does not use recurrent clip propagation or optical-flow-guided hidden-state transfer; its propagation is transformer-based and cross-domain, with the spatial output reused as the temporal value representation (Liang et al., 2022). Relative to PGDUDST, which is a single-frame CASSI reconstruction method based on proximal gradient descent unfolding and a Dense-spatial Spectral-attention Transformer prior, PG-SVRT moves from single-image spectral reconstruction to video-level SCI and directly optimizes temporal consistency through video-based feature propagation (Chen et al., 2023).

An important limitation explicitly stated in the paper is that DynaSpec remains controlled and idealized, with indoor lighting and specific motion statistics. Another reported limitation is that experiments use short sequences with $$Y_i(h, w) = \sum_{c=1}^{C} \Phi(h, w - \sigma(c)) \cdot X_i(h, w, c).$$8; long-sequence memory and streaming behavior are not explored in the main text. A plausible implication is that PG-SVRT, as reported, is best understood as a compact and efficient short-range video-level SCI reconstructor rather than a long-horizon recurrent spectral video model.

Within the reported evidence, PG-SVRT occupies a specific position: it combines degradation-aware input modeling through MGDP, efficient spatial-then-temporal propagation through CDPA, and low-cost cross-domain fusion through bridged tokens and MDFFN. Under the tested settings, especially with DD-CASSI, the paper reports the best balance of reconstruction quality, spectral accuracy, temporal coherence, and efficiency (Cai et al., 28 Feb 2026).