Spectral Diffusion Prior (SDP)

• Spectral Diffusion Prior (SDP) is a diffusion-derived prior that leverages clean, high-quality spectral data to guide inverse problem reconstruction and conditional generation.
• It employs variations of DDPM formulations and latent or covariance adaptations to capture spectral structures across hyperspectral imaging, speech synthesis, and CT modalities.
• Empirical results show significant gains in metrics like PSNR, SSIM, and correlation, validating SDP’s effectiveness in balancing reconstruction fidelity and computational efficiency.

Searching arXiv for papers on “Spectral Diffusion Prior” and related usages. Spectral Diffusion Prior (SDP) denotes a class of diffusion-derived priors for signals with spectral structure, used to regularize or guide inverse problems, restoration, and conditional generation. In the cited literature, the term is applied to several distinct objects: pixel-wise hyperspectral signatures, compact latent spectral codes, frequency-domain amplitude-and-phase representations, mel-spectrogram-conditioned Gaussian priors, multi-material spectral CT volumes, and anisotropic Gaussian priors with explicit spectral shaping. Despite this terminological variation, the recurring pattern is to learn a prior from clean, well-exposed, or otherwise high-quality data and then inject that prior into a downstream reconstruction or sampling procedure so that fine spectral-spatial detail is recovered more faithfully than with regression-only models or hand-crafted regularization alone (Liu et al., 2023, Wu et al., 2023, Li et al., 2024, Jiang et al., 28 Mar 2025, Alido et al., 15 May 2025, Yu et al., 18 Jul 2025).

1. Scope of the term

Across the cited literature, “spectral” does not have a single fixed meaning. In hyperspectral imaging it refers to wavelength-resolved spectra; in exposure correction it refers to amplitude and phase spectra in the frequency domain; in speech it refers to mel-spectrogram statistics; and in structured diffusion for imaging inverse problems it refers to frequency-dependent covariance shaping. Correspondingly, an SDP may be a learned DDPM prior, a latent conditional prior, an adaptive Gaussian prior, or a score-based prior embedded in a posterior sampler.

Setting	Prior variable	Primary use
HSI super-resolution (Liu et al., 2023)	Pixel-wise spectrum $x\in\mathbb R^{N_B}$	MAP regularizer for fusion
Snapshot spectral compressive imaging (Wu et al., 2023)	Latent $z_0\in\mathbb R^{N\times C}$	Prior-guided deep unfolding
Exposure correction (Li et al., 2024)	$Z\in\mathbb R^{1\times1\times M}$	Affine fusion in OS-SSM
Speech synthesis (Lee et al., 2021)	$q(z_T\mid c)=\mathcal N(0,\Sigma(c))$	Data-dependent adaptive prior
Volumetric spectral CT (Jiang et al., 28 Mar 2025)	Material volume $x\in\mathbb R^{M\times K}$	Posterior sampling with forward model
HSI reconstruction (Yu et al., 18 Jul 2025)	Compact spectral feature $x_0\in\mathbb R^d$	SPIM-based feature modulation
Ultra-low-dose spectral CT (Peng et al., 8 Feb 2026)	Full-spectrum prior image $x_F$ and its latent	Dual-domain latent diffusion
Hyperspectral unmixing (Zhu et al., 10 Dec 2025)	Endmember matrix $\mathbf A\in\mathbb R^{C\times K}$	Conditional posterior sampling

These usages place SDP at the intersection of generative modeling and physics-constrained inference. In some works the prior is the central estimator, while in others it acts as an auxiliary source of “degradation-free” information that steers a separate reconstruction backbone.

2. Core mathematical formulations

Many SDP instantiations adopt the standard DDPM forward process

$$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$

with closed form

$$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$

and a reverse model parameterized by a noise predictor $z_0\in\mathbb R^{N\times C}$0. This form appears for 1-D spectral signatures in fusion-based HSI super-resolution, for compact latent codes in snapshot spectral compressive imaging, for multi-material volumes in spectral CT, and for low-dimensional HSI features used as plug-in priors in reconstruction backbones (Liu et al., 2023, Wu et al., 2023, Jiang et al., 28 Mar 2025, Yu et al., 18 Jul 2025).

Several variants modify either the prior distribution or the state space. PriorGrad replaces the terminal isotropic Gaussian with a conditioning-dependent Gaussian,

$z_0\in\mathbb R^{N\times C}$1

where $z_0\in\mathbb R^{N\times C}$2 is diagonal and derived from the mel-spectrogram. OSMamba places the process in a $z_0\in\mathbb R^{N\times C}$3 latent and conditions denoising on $z_0\in\mathbb R^{N\times C}$4, using

$z_0\in\mathbb R^{N\times C}$5

with $z_0\in\mathbb R^{N\times C}$6 steps and a teacher-student prior distillation objective (Lee et al., 2021, Li et al., 2024).

A more general reformulation appears in Whitened Score diffusion, which replaces isotropic noising by an anisotropic SDE

$z_0\in\mathbb R^{N\times C}$7

and learns the whitened score $z_0\in\mathbb R^{N\times C}$8 rather than the standard score. This avoids explicit covariance inversion and induces a prior with frequency-dependent variance in the Fourier basis, making the “spectral” structure an attribute of the forward covariance operator itself (Alido et al., 15 May 2025).

FSP-Diff introduces yet another specialization: a full-spectrum prior formed by fusing energy-bin projections in the log-domain,

$z_0\in\mathbb R^{N\times C}$9

and then treating $Z\in\mathbb R^{1\times1\times M}$0 as a Gaussian tethering prior around which the energy-bin image is conditioned in latent diffusion (Peng et al., 8 Feb 2026).

3. Modes of integration into inverse problems

In the most explicit probabilistic formulation, SDP enters as a regularizer in a maximum a posteriori objective. For fusion-based HSI super-resolution, the degradation model

$Z\in\mathbb R^{1\times1\times M}$1

is combined with a diffusion-derived penalty obtained by retaining transition information between neighboring reverse states. The resulting SDP-MAP objective adds the sum of denoising losses over all pixels and timesteps, and the optimization is solved sequentially from $Z\in\mathbb R^{1\times1\times M}$2 down to $Z\in\mathbb R^{1\times1\times M}$3 with Adam (Liu et al., 2023).

A different integration strategy appears in snapshot spectral compressive imaging. There, deep unfolding alternates a physics-driven projection

$Z\in\mathbb R^{1\times1\times M}$4

with a learned denoiser. SDP upgrades each stage to a Trident Transformer that fuses spatial flow, cross-spectral flow, and cross-prior flow. In the cross-prior branch, the regenerated latent prior $Z\in\mathbb R^{1\times1\times M}$5 supplies keys and values in

$Z\in\mathbb R^{1\times1\times M}$6

so that clean-image structure is injected into the unfolding denoiser (Wu et al., 2023).

OSMamba embeds its SDP in an Omnidirectional Spectral State Space Block. After OS-Scan and S6 processing, the sampled prior $Z\in\mathbb R^{1\times1\times M}$7 is linearly expanded into $Z\in\mathbb R^{1\times1\times M}$8 and fused as

$Z\in\mathbb R^{1\times1\times M}$9

This is an affine modulation mechanism in the spectral domain, intended to bias amplitude-phase features toward globally coherent, well-exposed structures (Li et al., 2024).

In plug-in HSI reconstruction, the Spectral Prior Injector Module (SPIM) performs a closely related gating-and-shift operation. With feature map $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$0 and spectral prior $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$1,

$q(z_T\mid c)=\mathcal N(0,\Sigma(c))$2

The same prior vector can be injected after multiple Transformer or convolutional stages. FSP-Diff likewise uses conditioning rather than explicit regularization: projection-domain diffusion first denoises each noisy energy-bin projection, then image-domain diffusion fuses three streams, $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$3, $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$4, and $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$5, where $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$6 is the full-spectrum prior reconstruction (Yu et al., 18 Jul 2025, Peng et al., 8 Feb 2026).

Posterior-sampling formulations integrate SDP with an explicit physical forward model. In volumetric spectral CT, Jiang et al. use the learned prior term $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$7 inside a Spectral DPS sampler targeting

$q(z_T\mid c)=\mathcal N(0,\Sigma(c))$8

with the polychromatic forward model $q(z_T\mid c)=\mathcal N(0,\Sigma(c))$9. The discrete algorithm alternates a diffusion step and an MBIR step with a compressed forward model and a TV penalty along $x\in\mathbb R^{M\times K}$0. DPS4Un for semiblind unmixing similarly treats a pretrained conditional spectrum diffusion model as a posterior sampler, combining the learned endmember prior with superpixel-based data fidelity and iterative abundance updates (Jiang et al., 28 Mar 2025, Zhu et al., 10 Dec 2025).

4. Architectures, conditioning mechanisms, and efficiency

A prominent design trend is to shift diffusion away from full-resolution tensors and into compact spectral or latent spaces. In snapshot spectral compressive imaging, a lightweight encoder maps $x\in\mathbb R^{M\times K}$1 to $x\in\mathbb R^{M\times K}$2 with $x\in\mathbb R^{M\times K}$3 tokens and $x\in\mathbb R^{M\times K}$4 channels. Because diffusion runs entirely in this low-dimensional latent space, the method reports two orders of magnitude savings in memory and uses $x\in\mathbb R^{M\times K}$5 steps; per-sample FLOPs are $x\in\mathbb R^{M\times K}$6 for $x\in\mathbb R^{M\times K}$7. OSMamba likewise uses a compact $x\in\mathbb R^{M\times K}$8 latent with $x\in\mathbb R^{M\times K}$9, a two-layer ReLU MLP of width $x_0\in\mathbb R^d$0, and $x_0\in\mathbb R^d$1 reverse steps. The plug-in HSI reconstruction SDP also uses a shallow three-layer MLP denoiser, a lightweight HSI Feature Extractor, $x_0\in\mathbb R^d$2, a 5-epoch diffusion warm-up, and Stage II training for 50 epochs total (Wu et al., 2023, Li et al., 2024, Yu et al., 18 Jul 2025).

Efficiency pressures are especially visible in spectral CT. A full 3D score network in Spectral DPS would require $x_0\in\mathbb R^d$3 GB of GPU memory, so the prior is trained as a 2D U-Net-style denoiser and applied slice-by-slice, while inter-slice continuity is enforced by $x_0\in\mathbb R^d$4 with $x_0\in\mathbb R^d$5. The forward model is further compressed to $x_0\in\mathbb R^d$6 energy bins. FSP-Diff uses a dual-domain latent diffusion design with only $x_0\in\mathbb R^d$7 diffusion steps per stage and reports reconstruction of each bin in $x_0\in\mathbb R^d$8 s on an NVIDIA V100 (Jiang et al., 28 Mar 2025, Peng et al., 8 Feb 2026).

Conditioning strategies also vary sharply. PriorGrad computes $x_0\in\mathbb R^d$9 from frame-wise mel energy and uses the same covariance both in training and sampling; OSMamba conditions on a learned vector $x_F$0 from a frequency-domain extractor; snapshot SCI conditions on latent measurement features $x_F$1; and Whitened Score diffusion encodes spectral inductive bias through the noising operator $x_F$2 itself rather than through a conventional condition encoder. This suggests that SDP can be instantiated either as a learned latent descriptor, as a covariance adaptation, or as a structural property of the diffusion process (Lee et al., 2021, Alido et al., 15 May 2025).

5. Empirical behavior across domains

On snapshot spectral compressive imaging, the latent-diffusion-enhanced deep unfolding model achieves PSNR $x_F$3 and SSIM $x_F$4 on the synthetic KAIST setting with 28 bands and $x_F$5 stages, compared with $x_F$6 and $x_F$7 for RDLUF-MixS2. In a bird-wing patch, the local spectral curve reaches correlation $x_F$8 with ground truth. Ablation shows that removing the diffusion prior drops PSNR by $x_F$9, and eliminating the Trident Transformer loses another $\mathbf A\in\mathbb R^{C\times K}$0. In the plug-in HSI reconstruction setting, MST-S improves from $\mathbf A\in\mathbb R^{C\times K}$1 to $\mathbf A\in\mathbb R^{C\times K}$2, and BiSRNet improves from $\mathbf A\in\mathbb R^{C\times K}$3 to $\mathbf A\in\mathbb R^{C\times K}$4; average SSIM rises by $\mathbf A\in\mathbb R^{C\times K}$5 to $\mathbf A\in\mathbb R^{C\times K}$6 (Wu et al., 2023, Yu et al., 18 Jul 2025).

For fusion-based HSI super-resolution, SDP yields strong gains on three synthetic benchmarks. On PaviaU, it reports PSNR $\mathbf A\in\mathbb R^{C\times K}$7, SAM $\mathbf A\in\mathbb R^{C\times K}$8, RMSE $\mathbf A\in\mathbb R^{C\times K}$9, ERGAS $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$0, and UIQI $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$1. On KSC, the reported values are PSNR $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$2, SAM $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$3, RMSE $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$4, ERGAS $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$5, and UIQI $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$6. On DC, the reported values are PSNR $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$7, SAM $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$8, RMSE $$q(x_t\mid x_{t-1})=\mathcal N\!\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$9, ERGAS $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$0, and UIQI $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$1. For real Hyperion–Sentinel data, the no-reference metrics are $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$2, $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$3, and QNR $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$4 (Liu et al., 2023).

In speech synthesis, the adaptive spectral prior of PriorGrad improves convergence and final quality relative to a standard conditional diffusion baseline. Reported full-convergence metrics are LS-MAE $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$5 vs. $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$6, MR-STFT $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$7 vs. $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$8, MCD $$q(x_t\mid x_0)=\mathcal N\!\bigl(x_t;\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$9 vs. $z_0\in\mathbb R^{N\times C}$00, F0 RMSE $z_0\in\mathbb R^{N\times C}$01 vs. $z_0\in\mathbb R^{N\times C}$02, and Sinkhorn divergence $z_0\in\mathbb R^{N\times C}$03 vs. $z_0\in\mathbb R^{N\times C}$04. Subjective MOS at $z_0\in\mathbb R^{N\times C}$05 is $z_0\in\mathbb R^{N\times C}$06 vs. $z_0\in\mathbb R^{N\times C}$07, and at $z_0\in\mathbb R^{N\times C}$08 the gap is $z_0\in\mathbb R^{N\times C}$09 vs. $z_0\in\mathbb R^{N\times C}$10. Whitened Score diffusion reports consistent $z_0\in\mathbb R^{N\times C}$11–$z_0\in\mathbb R^{N\times C}$12 dB PSNR gains over isotropic diffusion priors, including CIFAR improvements from $z_0\in\mathbb R^{N\times C}$13 dB to $z_0\in\mathbb R^{N\times C}$14 dB and CelebA improvements from $z_0\in\mathbb R^{N\times C}$15 dB to $z_0\in\mathbb R^{N\times C}$16 dB at SNR $z_0\in\mathbb R^{N\times C}$17. OSMamba reports state-of-the-art quantitative and qualitative performance on multiple-exposure and mixed-exposure datasets, with its SDP framed as a degradation-free diffusion prior for severely under- and over-exposed regions (Lee et al., 2021, Li et al., 2024, Alido et al., 15 May 2025).

In spectral CT and unmixing, the benefits are tied not only to distortion metrics but also to physically meaningful reconstruction properties. Spectral DPS is reported to outperform InceptNet and conditional DDPM in contrast quantification, inter-slice continuity, and resolution preservation for volumetric material decomposition. DPS4Un achieves the lowest aRMSE on Jasper Ridge, $z_0\in\mathbb R^{N\times C}$18 vs. the nearest $z_0\in\mathbb R^{N\times C}$19, and reports aSAD $z_0\in\mathbb R^{N\times C}$20; on Urban it reports aRMSE $z_0\in\mathbb R^{N\times C}$21 and aSAD $z_0\in\mathbb R^{N\times C}$22; on SMScene it reports aRMSE $z_0\in\mathbb R^{N\times C}$23 and the best aSAD $z_0\in\mathbb R^{N\times C}$24. In ultra-low-dose spectral CT, the isolated effect of the full-spectrum prior is quantified by the IP $z_0\in\mathbb R^{N\times C}$25 FSP-Diff ablation: average PSNR gain is $z_0\in\mathbb R^{N\times C}$26 dB and SSIM gain is $z_0\in\mathbb R^{N\times C}$27 to $z_0\in\mathbb R^{N\times C}$28, with per-bin PSNR reaching $z_0\in\mathbb R^{N\times C}$29 and SSIM reaching $z_0\in\mathbb R^{N\times C}$30 (Jiang et al., 28 Mar 2025, Zhu et al., 10 Dec 2025, Peng et al., 8 Feb 2026).

6. Conceptual issues, limitations, and research directions

A common misconception is that SDP refers to a single architecture. The cited works use the label for at least four different constructions: a DDPM over spectral signatures, a compact latent prior regenerated from measurements, a conditioning-dependent Gaussian prior, and a structured score prior induced by anisotropic covariance. A second misconception is that “spectral” always refers to wavelength bands. In fact, the literature includes pixel spectra in hyperspectral imaging, amplitude-phase spectra in frequency-domain restoration, mel-spectrogram statistics in speech, and Fourier-domain spectral shaping through $z_0\in\mathbb R^{N\times C}$31 in structured diffusion. The term is therefore best understood as domain-dependent rather than universally standardized (Liu et al., 2023, Li et al., 2024, Lee et al., 2021, Alido et al., 15 May 2025).

Several limitations recur. The snapshot SCI work explicitly identifies the large computational cost challenge in LDM and addresses it by moving to a lightweight latent design. The plug-in HSI reconstruction work notes that two-stage training adds complexity and hyper-parameters, and that SDP alone does not exploit spatial context. Volumetric spectral CT must resort to slice-by-slice diffusion because a full 3D score network would require $z_0\in\mathbb R^{N\times C}$32 GB of GPU memory. FSP-Diff addresses high-dimensional spectral data by compact latent embedding and only $z_0\in\mathbb R^{N\times C}$33 steps per stage. Proposed extensions include combining spectral diffusion with a 2D or 3D U-Net, applying the paradigm to other compressive imaging problems such as ToF and MRI, increasing $z_0\in\mathbb R^{N\times C}$34, and conditioning the diffusion model more tightly in an end-to-end fashion. This suggests that future SDP research will continue to trade off prior expressivity, conditioning fidelity, physical-model integration, and computational tractability (Wu et al., 2023, Jiang et al., 28 Mar 2025, Yu et al., 18 Jul 2025, Peng et al., 8 Feb 2026).