---
title: Spectral Unobservable Feature Autoencoder
url: https://www.emergentmind.com/topics/spectral-unobservable-feature-autoencoder-speuae
type: topic
---

# Spectral Unobservable Feature Autoencoder

Searching arXiv for the cited papers to ground the article.
{"query":"id:2309.14286 OR id:2507.12967","max_results":5}
I found the relevant arXiv papers and will use them as the basis for the article: "Coherent Spectral Feature Extraction Using Symmetric Autoencoders" [2309.14286] and "RGB Pre-Training Enhanced Unobservable Feature Latent Diffusion Model for Spectral Reconstruction" [2507.12967].
The Spectral Unobservable Feature Autoencoder (SpeUAE) is a spectral autoencoder introduced for spectral reconstruction (SR) from RGB images. Its central purpose is to extract, compress, and represent the spectral information of a hyperspectral image (HSI) that is not recoverable from the corresponding RGB image alone, and to express that information as a 3-channel unobservable feature manifold aligned with RGB space. In the two-stage framework that introduces it, SpeUAE serves as the first-stage spectral encoder–decoder: it maps a full HSI \(I^\text{HS}\) to a learned unobservable feature \(F^\text{Un} \in \mathbb{R}^{3 \times N_x \times N_y}\), and reconstructs the HSI from \(F^\text{Un}\) together with the RGB image \(I^\text{RGB}\) [2507.12967].

## 1. Problem setting and definition of the unobservable feature

Spectral reconstruction considers an HSI
\[
I^\text{HS} \in \mathbb{R}^{B \times N_x \times N_y}
\]
and a corresponding RGB image
\[
I^\text{RGB} \in \mathbb{R}^{3 \times N_x \times N_y},
\]
with the RGB camera modeled by a spectral sensitivity function
\[
P \in \mathbb{R}^{3 \times B}.
\]
The imaging process is
\[
I^\text{RGB} = I^\text{HS} \times_1 P^T. \tag{1}
\]
Because typically \(B \gg 3\), the forward map is many-to-one: multiple spectra can generate the same RGB observation.

The formal observable–unobservable decomposition is defined through the singular value decomposition of \(P\). Let \(V_1 \in \mathbb{R}^{B \times 3}\) denote the right singular vectors corresponding to non-zero singular values, and \(V_0 \in \mathbb{R}^{B \times (B-3)}\) denote the right singular vectors spanning the null space of \(P\). Then any HSI can be written as
\[
I^\text{HS} = F^\text{Ob} \times_1 V_1^T + F^\text{Un} \times_1 V_0^T, \tag{2}
\]
with
\[
F^\text{Ob} = I^\text{HS} \times_1 V_1 \in \mathbb{R}^{3 \times N_x \times N_y},
\]
\[
F^\text{Un} = I^\text{HS} \times_1 V_0 \in \mathbb{R}^{(B-3) \times N_x \times N_y}.
\]
Here \(F^\text{Ob}\) is the observable feature, while \(F^\text{Un}\) is the component invisible to the RGB sensor. Since \(P V_0 = \mathbf{0}\),
\[
I^\text{RGB} = F^\text{Ob} \times_1 V_1^T P^T, \tag{3}
\]
so the RGB image depends only on the observable component. The solution space of spectral reconstruction can therefore be expressed as
\[
\hat I^\text{HS} = I^\text{RGB} \times_1 (V_1^T P^T)^{-1} V_1^T + \zeta \times_1 V_0^T, \tag{4}
\]
where \(\zeta \in \mathbb{R}^{(B-3)\times N_x \times N_y}\) is arbitrary. In this formulation, accurate SR requires selecting a plausible unobservable component according to a learned conditional distribution \(p(F^\text{Un} \mid I^\text{RGB})\) [2507.12967].

A common misconception is to equate SpeUAE’s learned representation directly with the exact null-space coordinates \(I^\text{HS} \times_1 V_0\). In the proposed method, that exact SVD-based decomposition is used to motivate the problem, but the practical system does not explicitly compute unobservable features through SVD. Instead, SpeUAE learns an effective 3-channel representation that is interpreted as a compressed unobservable feature.

## 2. SpeUAE architecture and training objective

SpeUAE consists of a spectral encoder \(E_\theta\) and a spectral decoder \(D_\vartheta\). The encoder maps the HSI to a learned unobservable feature,
\[
F^\text{Un} = E_\theta(I^\text{HS}), \tag{5}
\]
and the decoder reconstructs the HSI from the learned unobservable feature and the RGB image,
\[
\hat I^\text{HS} = D_\vartheta(F^\text{Un}, I^\text{RGB}). \tag{6}
\]
The encoder and decoder operate spectrally, pixel-by-pixel, without spatial coupling. They are implemented as MLP stacks applied independently per pixel spectrum. The spectral encoder and decoder of SpeUAE have a hidden dimension of 64. The encoder maps a \(B\)-dimensional spectrum to a 3-dimensional code, so the output forms a 3-channel tensor \(F^\text{Un} \in \mathbb{R}^{3 \times N_x \times N_y}\). The decoder consumes the corresponding 3-dimensional unobservable code together with the 3-dimensional RGB vector for each pixel and reconstructs the full spectrum.

The decoder uses the RGB image as an explicit carrier of observable information. This is not a minor implementation convenience but a structural assumption: the observable component is treated as already encapsulated by \(I^\text{RGB}\), and the encoder is trained to compress the residual spectral information needed beyond RGB.

SpeUAE is optimized with two mean-squared-error terms. The first is the HSI reconstruction loss,
\[
\mathcal{L}_\text{HSI-Re} = \mathbb{E}_{(I^\text{HS}, I^\text{RGB})} \bigl\| I^\text{HS} - \hat I^\text{HS} \bigr\|_2^2. \tag{9}
\]
The second is an alignment loss defined through a frozen spatial autoencoder (SpaAE) inherited from an RGB-pretrained latent diffusion model:
\[
Z^\text{Un} = E_\text{spatial}(F^\text{Un}), \tag{7}
\]
\[
\hat F^\text{Un} = D_\text{spatial}(Z^\text{Un}), \tag{8}
\]
\[
\mathcal{L}_\text{align}
= \mathbb{E}_{F^\text{Un}} \bigl\| D_\text{spatial}(E_\text{spatial}(F^\text{Un})) - F^\text{Un} \bigr\|_2^2. \tag{10}
\]
The total objective is
\[
\mathcal{L}_\text{SpeUAE} = \mathcal{L}_\text{HSI-Re} + \lambda \mathcal{L}_\text{align}, \tag{11}
\]
with \(\lambda = 0.1\) reported as the best setting in the ablation study. No adversarial term and no KL term are used; SpeUAE is a deterministic autoencoder trained with MSE losses [2507.12967].

The role of the alignment term is specific. It does not assert that the learned unobservable feature is physically equivalent to an RGB image. Rather, it enforces that the 3-channel manifold behaves like an RGB image from the perspective of the frozen SpaAE, so that the spatial latent machinery of the pretrained RGB model can be reused in the next stage.

## 3. Position within the ULDM pipeline

SpeUAE is the first stage of a two-stage framework that transforms an RGB-pretrained latent diffusion model (RGB-LDM) into an unobservable feature latent diffusion model (ULDM). In the first stage, SpeUAE learns spectral structure representation. In the second stage, the learned unobservable feature is passed through the spatial autoencoder and modeled by latent diffusion conditioned on the corresponding RGB latent [2507.12967].

The overall workflow can be summarized as follows.

| Component | Mapping | Role |
|---|---|---|
| SpeUAE encoder | \(I^\text{HS} \mapsto F^\text{Un}\) | Extracts 3-channel unobservable feature |
| SpaAE | \(F^\text{Un} \mapsto Z^\text{Un}\), \(I^\text{RGB} \mapsto Z^\text{RGB}\) | Provides shared spatial latent space |
| ULDM | \(p(Z^\text{Un}\mid Z^\text{RGB})\) | Models conditional distribution of unobservable latent |

With SpeUAE fixed after stage 1, the latent embedding of the unobservable feature is
\[
Z^\text{Un} = E_\text{spatial}(F^\text{Un}) = E_\text{spatial}(E_\theta(I^\text{HS})), \tag{12}
\]
and the RGB latent is
\[
Z^\text{RGB} = E_\text{spatial}(I^\text{RGB}). \tag{13}
\]
ULDM is then obtained by fine-tuning an RGB-LDM initialized from Stable Diffusion 2.1-v to model the conditional distribution \(p(Z^\text{Un}\mid Z^\text{RGB})\).

The forward diffusion process for \(Z_0 := Z^\text{Un}\) is
\[
Z_t = \sqrt{\bar \alpha_t} Z_0 + \sqrt{1 - \bar \alpha_t} \epsilon, \tag{14}
\]
where \(\epsilon \sim \mathcal{N}(0, I)\), \(\bar \alpha_t = \prod_{s=0}^t (1 - \beta_s)\), and \(T=1000\) diffusion steps are used in training. The reverse process is parameterized by a conditional denoising network \(\epsilon_\phi\), trained with
\[
\mathcal{L}_\text{LDM} = \mathbb{E}_{(Z_0, Z^\text{RGB}), \epsilon, t} \bigl\| \epsilon - \hat \epsilon \bigr\|_2^2, \tag{15}
\]
where \(\hat \epsilon = \epsilon_\phi(Z_t; Z^\text{RGB}, t)\).

At inference, only the RGB image is given. It is encoded to \(Z^\text{RGB}\), the ULDM samples \(Z^\text{Un} \sim p(Z^\text{Un}\mid Z^\text{RGB})\), the SpaAE decodes \(Z^\text{Un}\) to \(F^\text{Un}\), and the SpeUAE decoder reconstructs the HSI:
\[
\hat I^\text{HS} = D_\vartheta(F^\text{Un}, I^\text{RGB}).
\]
Sampling uses DDIM in 20 steps for efficiency rather than the full 1000-step training chain.

## 4. Relation to latent-factor autoencoding and SymAE

Although SpeUAE is introduced for RGB-to-HSI reconstruction, it sits within a broader family of hyperspectral autoencoders that attempt to separate latent spectral factors. A useful precursor is the symmetric autoencoder (SymAE), which was designed for hyperspectral feature extraction by disentangling class-invariant coherent features from nuisance variability on a pixel-by-pixel basis [2309.14286].

SymAE assumes a generative relation
\[
P = f(C, N),
\]
where \(C \in \mathbb{R}^{d_c}\) denotes coherent latent features shared within a group, and \(N \in \mathbb{R}^{d_n}\) denotes nuisance latent features that vary per pixel. Its coherent encoder is permutation invariant over a group of pixels:
\[
C = CEnc_\text{mean}(X)
= \frac{1}{n_\tau} \sum_{\tau=1}^{n_\tau} CEnc(X[\tau]), \tag{2}
\]
so \(C\) is unchanged under permutations of the group. The nuisance path is pixel-specific and regularized by dropout:
\[
N[\tau] = \mathtt{Dropout}(NEnc(X[\tau])), \tag{3}
\]
with dropout probability \(p=0.5\) during training. Reconstruction is performed from the concatenated code \([C, N[\tau]]\), and training uses only MSE reconstruction loss, without explicit disentanglement penalties.

The relevance of this earlier architecture to SpeUAE is conceptual rather than architectural identity. SymAE addresses latent separation within hyperspectral data by forcing shared structure into a reliable, permutation-invariant path and variability into a stochastic path. SpeUAE addresses a different decomposition: observable spectral information already encoded by RGB versus residual spectral information that is unobservable to the RGB sensor. This suggests a broader interpretation in which hyperspectral autoencoders can be organized by the latent factors they aim to isolate: coherent versus nuisance in SymAE, and observable versus unobservable in SpeUAE. The shared methodological theme is that disentanglement is induced structurally rather than through explicit generative priors or handcrafted physical noise models.

## 5. Data, ablations, and empirical behavior

SpeUAE is trained on paired RGB–HSI data from CAVE, ICVL, and NTIRE22. CAVE contains 32 HSIs of size \(512 \times 512\) with 31 wavelengths from 400–700 nm, using 26 for training and 6 for testing. ICVL contains 201 scenes, each with a 31-band HSI of size \(1392 \times 1300\), using 107 for training and 30 for testing. NTIRE22 contains 1000 HSIs of size \(482 \times 512\) with 31 wavelengths, using 900 for training, 50 for validation, and 50 for test, although only the validation set is publicly accessible. RGB images are synthesized from HSIs using camera SSFs: Nikon D700 SSF for CAVE and ICVL, and the challenge-provided SSFs for NTIRE22. HSIs are cropped to \(128 \times 128\) patches with stride 64, and random rotation and flipping are used as augmentation [2507.12967].

The optimization details for SpeUAE are specific: Adam with initial learning rate \(4 \times 10^{-4}\), batch size 20, 100k iterations, hidden dimension 64, alignment weight \(\lambda = 0.1\), and frozen SpaAE parameters during SpeUAE training.

The central ablation isolates the effect of separating the unobservable feature and constraining it to a 3D manifold. On NTIRE22, the following variants are reported:

| Variant | Configuration | NTIRE22 result |
|---|---|---|
| Pixel DM | no AE, no separate unobservable | PSNR 32.65, SSIM 0.9223, SAM 16.54 |
| LDM (no separation) | AE, no separate unobservable, 3D | PSNR 33.47, SSIM 0.9059, SAM 7.081 |
| Ours (SpeUAE, 3D) | AE, separate unobservable, 3D | PSNR 42.26, SSIM 0.9888, SAM 4.355 |
| ULDM (6D) | AE, separate unobservable, 6D | PSNR 41.34, SSIM 0.9890, SAM 4.699 |

These results show that using a latent autoencoder without explicit unobservable-feature separation yields only modest gain over pixel-space diffusion, whereas separating the unobservable feature with SpeUAE and representing it in 3D produces a large improvement. Increasing the latent dimensionality from 3D to 6D slightly degrades performance, indicating that, in the reported setup, a 3D manifold is not merely sufficient but more effective than a higher-dimensional alternative.

The alignment loss ablation on NTIRE22 reports:
- \(\lambda = 0\): PSNR 41.84, SSIM 0.9895, SAM 4.638;
- \(\lambda = 0.1\): PSNR 42.26, SSIM 0.9890, SAM 4.355;
- \(\lambda = 0.2\): PSNR 41.77, SSIM 0.9889, SAM 4.754.

A moderate alignment weight therefore improves PSNR and SAM relative to no alignment, while stronger regularization slightly harms performance. This indicates that soft manifold alignment is beneficial, but over-constraining the learned spectral manifold can reduce reconstruction quality.

In full SR evaluation, the method combining SpeUAE and ULDM is reported as competitive or superior to strong baselines including HSCNN+, HDNet, MST, MST++, R2HGAN, and R2H-CCD. On ICVL it reports PSNR 47.94 dB, SSIM 0.9992, and SAM 1.176, compared with MST++ at 47.50 dB, 0.9987, and 1.242, and R2HGAN at 45.27 dB, 0.9979, and 1.453. On NTIRE22 it reports PSNR 42.26 dB, SSIM 0.9888, and SAM 4.355; MST++ attains 41.16 dB, 0.9870, and 3.691, while R2HGAN attains 41.18 dB, 0.9882, and 4.310. The same framework is also evaluated on relighting, where HSI-based relighting using the reconstructed spectra achieves the highest PSNR in most cases, including 48.28 dB under CIE A on CAVE.

## 6. Limitations, clarifications, and likely extensions

Several limitations follow directly from the reported formulation and experiments [2507.12967]. SpeUAE depends on the training data distribution and on the camera SSF used to generate RGB images. Its generalization to different sensors or substantially different spectral distributions is therefore not guaranteed by the presented setup. The complete framework is computationally heavy, involving 100k SpeUAE iterations, 300 epochs of ULDM training, and 6×V100 GPUs for the diffusion stage. The alignment mechanism is heuristic in the sense that it is implemented as a simple MSE reconstruction penalty through the frozen SpaAE, rather than through a more elaborate manifold-matching objective. The theoretical analysis accompanying ULDM also relies on low-dimensional linear manifold assumptions for spectra and sparsity assumptions for HSIs.

A second clarification concerns the semantics of the learned latent. SpeUAE does not claim to recover a uniquely identifiable physical null-space decomposition for each scene. The learned \(F^\text{Un}\) is an effective compressed representation whose adequacy is operationally established by HSI reconstruction, SpaAE compatibility, diffusion modeling, and downstream relighting performance. It should therefore be understood as a task-oriented latent variable, not as an exact analytic inversion of the sensor model.

Potential extensions are stated or implied in the reported discussion. They include more expressive spectral encoders, incorporation of prior knowledge about material spectra or physics-based constraints, adaptive latent dimensionality beyond a fixed 3D or 6D manifold, improved generalization across sensors and SSFs, and evaluation on broader downstream tasks such as material classification, depth estimation, and 3D reconstruction. A plausible implication is that SpeUAE’s most distinctive contribution is not only compression, but the conversion of a high-dimensional spectral ambiguity into a form that can be handled by RGB-pretrained spatial generative models. In that sense, the autoencoder functions as an interface between hyperspectral inverse problems and latent diffusion architectures originally developed for natural RGB imagery.

Source: https://www.emergentmind.com/topics/spectral-unobservable-feature-autoencoder-speuae