Spectral-Spatial Tri-plane Aggregation (SSTA)

• Spectral-Spatial Tri-plane Aggregation (SSTA) is a network design that decomposes high-dimensional spectral and angular data into two sets of three learned 2D feature planes.
• It enables efficient spectral BRDF generation by fusing RGB and spectral data through bilinear sampling and adaptive feature fusion, reducing memory demands.
• SSTA’s concept generalizes to hyperspectral tasks via tri-axis attention, improving performance and generalization under challenging non-overlapping sampling conditions.

Spectral-Spatial Tri-plane Aggregation (SSTA) is a representation and network design for modeling coupled spectral and spatial structure through axis- or plane-wise decomposition rather than monolithic dense processing. In its exact published usage, SSTA denotes the core module of SpecGen, where a spectral bidirectional reflectance distribution function (BRDF) is represented as six learned 2D feature planes spanning angular and wavelength coordinates, enabling spectral BRDF generation from a single RGB sphere image under severe spectral-data scarcity (Jin et al., 24 Aug 2025). In a broader interpretive sense, the term can also describe the structural logic of the Spectral-spatial Axial Aggregation Transformer (SaaFormer) for hyperspectral image classification, whose multi-level spectral partitioning and dual spatial axial attention operate over one spectral axis and two spatial axes, although that work does not literally use the name “Spectral-Spatial Tri-plane Aggregation” (Zhao et al., 2023).

1. Terminological scope and historical placement

The exact phrase “Spectral-Spatial Tri-plane Aggregation” appears in SpecGen, where it names a network that models reflectance responses across wavelengths and incident-outgoing directions for spectral BRDF generation from a single RGB image of a sphere (Jin et al., 24 Aug 2025). In that setting, SSTA is not a generic synonym for any spectral-spatial model; it is a specific tri-plane factorization and feature aggregation mechanism embedded in the SpecGen pipeline.

A separate but related line of work appears in SaaFormer for hyperspectral image classification. That paper introduces a “spectral-spatial axial aggregation transformer” and argues that existing pixel-wise random sampling protocols in hyperspectral image classification create strong overlap between training and test patches, causing inflated accuracy and weak generalization under non-overlapping partitions (Zhao et al., 2023). The paper does not use the term SSTA. However, the design combines multi-level spectral partitioning with horizontal and vertical axial attention, so a plausible interpretation is that it realizes a tri-axis spectral-spatial aggregation pattern: one spectral axis and two spatial axes.

This distinction is important because the same phrase can otherwise be misunderstood as a standardized architecture family. The available literature instead supports a narrower conclusion: SSTA is an explicit term in spectral BRDF modeling, whereas its use for hyperspectral classification is conceptual and editorial rather than terminological.

2. Formal representation of SSTA in spectral BRDF generation

In SpecGen, the target object is a spectral BRDF treated as a continuous 4D function

$r = f(\theta_h,\theta_d,\varphi_d,\lambda),$

where $\theta_h$, $\theta_d$, and $\varphi_d$ are the Rusinkiewicz BRDF parameters and $\lambda$ is wavelength (Jin et al., 24 Aug 2025). The half-vector is defined by

$$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$

with $\theta_h$ the polar angle of $\boldsymbol{h}$ with respect to $\boldsymbol{n}=(0,0,1)$, $\theta_d$ the polar angle between $\theta_h$0 and $\theta_h$1, and $\theta_h$2 the azimuth around $\theta_h$3.

The motivation for the tri-plane decomposition is computational and statistical. A dense 4D grid has memory scaling $\theta_h$4 for resolution $\theta_h$5 per dimension, while a single large coordinate MLP entangles angular and spectral behavior and provides no natural mechanism for asymmetric supervision from RGB-only and spectral data. SpecGen therefore adopts a K-Planes-inspired factorization into six learned 2D planes: three angular “spatial tri-planes” and three angle-wavelength “spectral tri-planes.” The angular planes are

• $\theta_h$6,
• $\theta_h$7,
• $\theta_h$8,

and the spectral planes are

• $\theta_h$9,
• $\theta_d$0,
• $\theta_d$1.

For a query $\theta_d$2, the model samples one feature vector from each plane by bilinear interpolation: $\theta_d$3

$\theta_d$4

This decomposition is designed to separate angular structure, which can be shared with RGB BRDF data, from wavelength-dependent variation, which must be learned from scarce spectral BRDF measurements. The implementation initializes the tri-plane dimensions as $\theta_d$5, corresponding to $\theta_d$6 bins for $\theta_d$7, $\theta_d$8 bins for $\theta_d$9, $\varphi_d$0 bins for $\varphi_d$1, and $\varphi_d$2 wavelength channels obtained by subsampling the original $\varphi_d$3 spectral channels.

3. Network architecture and aggregation mechanism in SpecGen

Within SpecGen, SSTA is the core representation module that maps an input RGB sphere image to six 2D feature planes through a CNN encoder-decoder, after which queried BRDF coordinates are projected onto the planes and fused before scalar reflectance prediction (Jin et al., 24 Aug 2025). The pipeline is: input RGB sphere image; SSTA encoder-decoder producing three spatial tri-planes and three spectral tri-planes; coordinate projection and feature sampling; Adaptive Feature Fusion (AFF); a 3-layer BRDF mapping MLP; and reconstruction of the full spectral BRDF by repeated querying.

AFF replaces the fixed fusion commonly used in K-Planes-style models. The six sampled feature vectors are first transformed and combined: $\varphi_d$4 followed by global average pooling

$\varphi_d$5

a bottleneck transformation

$\varphi_d$6

branch-specific convolutions

$\varphi_d$7

softmax-based weights

$\varphi_d$8

and the weighted fusion

$\varphi_d$9

The final spectral reflectance prediction is

$\lambda$0

The architectural significance of SSTA is that it imposes an explicit separation between angular-only and angular-spectral interactions. This creates a direct interface for transferring supervision from RGB BRDF corpora to spectral BRDF generation, a property that a dense 4D tensor or a single entangled coordinate network does not provide naturally.

4. Tri-axis spectral-spatial aggregation in SaaFormer

SaaFormer addresses hyperspectral image classification under non-overlapping train-test partitions by combining multi-level spectral extraction with axial aggregation attention over the two spatial axes (Zhao et al., 2023). Hyperspectral images are represented as 3D cubes $\lambda$1, and a patch centered at a pixel is written as $\lambda$2, then linearly embedded to $\lambda$3.

Its first major component is multi-level spectral extraction. The feature tensor is divided along the spectral dimension into non-overlapping continuous spectral clips of different lengths $\lambda$4, preserving wavelength continuity within each clip. The model then introduces shifted spectral partitioning: after initial partitioning, halves of adjacent clips are recombined, for example

$\lambda$5

to create cross-partition interactions without abandoning local continuity. The paper further states that masking is used to prevent erroneous aggregation between physically unrelated first and last spectral segments after shifting.

Its second major component is axial aggregation attention over rows and columns. For $\lambda$6, standard linear projections form $\lambda$7, $\lambda$8, and $\lambda$9, after which max pooling along width and height produces row and column summaries. The basic aggregation attention is

$$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$0

and a positional-aware version adds learnable relative position embeddings along both axes. A parallel $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$1 convolution plus batch normalization preserves local spatial detail and is fused with the axial attention output.

Taken together, these elements support the tri-axis interpretation. The spectral dimension is processed through multi-level continuous clips and shifted partitions; the horizontal and vertical spatial dimensions are processed through row-wise and column-wise axial attention; and local spatial textures are preserved through the auxiliary convolutional path. This suggests that SaaFormer instantiates a spectral-spatial tri-plane logic in the sense of explicit aggregation over one spectral and two spatial axes, even though its published name remains “spectral-spatial axial aggregation transformer.”

5. Training strategies, supervision regimes, and quantitative behavior

In SpecGen, SSTA is designed for joint RGB-spectral training under spectral-data scarcity (Jin et al., 24 Aug 2025). The available spectral dataset is RGL with $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$2 isotropic materials, each with RGB and spectral BRDFs tabulated over $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$3 and $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$4 wavelengths; the RGB-only dataset is MERL with $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$5 RGB BRDFs. The reported split uses $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$6 spectral BRDF materials from RGL for training and $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$7 for testing. For training, $$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$8 BRDF points per material are sampled, approximately one tenth of the full angular-spectral volume.

The joint supervision mechanism is asymmetric. Spatial tri-planes are shared between RGB and spectral BRDFs, whereas spectral tri-planes are trained by spectral data and interact with spatial features through AFF. For spectral BRDFs, reflectance is first compressed with

$$\boldsymbol{h} = \frac{\boldsymbol{\omega_i} + \boldsymbol{\omega_o}}{\|\boldsymbol{\omega_i} + \boldsymbol{\omega_o}\|},$$9

and the spectral loss is

$\theta_h$0

augmented by total variation regularization: $\theta_h$1 For RGB BRDFs, spectral tri-plane features are averaged over wavelength and supervision is provided through

$\theta_h$2

The reported ablation indicates that training on spectral-only data (RGL-S) yields average PSNR $\theta_h$3 dB and SSIM $\theta_h$4 on $\theta_h$5 test materials, whereas joint training on RGL-C-S + MERL yields average PSNR $\theta_h$6 dB and SSIM $\theta_h$7. In downstream hyperspectral image reconstruction, SpecGen is compared with HD-Net and MST++, with average spectral rendering PSNR $\theta_h$8 dB, $\theta_h$9 dB, and $\boldsymbol{h}$0 dB respectively, and average SSIM $\boldsymbol{h}$1, $\boldsymbol{h}$2, and $\boldsymbol{h}$3. The paper summarizes the improvement as approximately $\boldsymbol{h}$4 dB in PSNR over strong hyperspectral reconstruction baselines.

SaaFormer uses a different supervision regime because its task is hyperspectral image classification rather than BRDF generation (Zhao et al., 2023). The reported training details include Adam, batch size $\boldsymbol{h}$5, initial learning rate $\boldsymbol{h}$6 decayed by $\boldsymbol{h}$7 at fixed epoch fractions, dropout $\boldsymbol{h}$8 after positional embeddings and in MLPs, and two encoder blocks each with four-head axial aggregation attention, MLP, and GELU. On Botswana, it reports OA/AA/$\boldsymbol{h}$9 of $\boldsymbol{n}=(0,0,1)$0 under random partition and $\boldsymbol{n}=(0,0,1)$1 under block-wise partition. On Pavia University, it reports $\boldsymbol{n}=(0,0,1)$2 under random partition and $\boldsymbol{n}=(0,0,1)$3 under block-wise partition. The paper emphasizes that block-wise sampling reveals substantial degradation in many baselines, whereas SaaFormer maintains smaller performance gaps.

6. Efficiency, misconceptions, and limitations

The principal efficiency claim behind SSTA-style decomposition is that structured factorization reduces the cost of representing or aggregating high-dimensional interactions without discarding global dependencies. In SpecGen, six 2D planes replace a dense 4D grid, yielding a memory pattern described as more efficient than $\boldsymbol{n}=(0,0,1)$4 volumetric storage, while still supporting continuous BRDF queries through bilinear sampling and a small MLP (Jin et al., 24 Aug 2025). In SaaFormer, axial attention avoids full 2D self-attention over an $\boldsymbol{n}=(0,0,1)$5 map: the paper contrasts $\boldsymbol{n}=(0,0,1)$6 full attention with row and column attention costs $\boldsymbol{n}=(0,0,1)$7 and $\boldsymbol{n}=(0,0,1)$8, for a total $\boldsymbol{n}=(0,0,1)$9 per spatial map (Zhao et al., 2023).

A common misconception is to treat “tri-plane” as referring only to the canonical three planes used in some neural field models. In SpecGen, the published SSTA decomposition actually uses six 2D planes: three angular planes and three angle-wavelength planes. The “tri-plane” wording refers to two coordinated triplets rather than a literal three-plane total. Another misconception is to assume that SSTA is already an established umbrella term across hyperspectral imaging and reflectance modeling. The evidence presently supports a more limited view: the exact name belongs to SpecGen, while its extension to SaaFormer is interpretive.

The published limitations are domain-specific. SpecGen notes sampling mismatch between training BRDF-space sampling and test-time illumination distributions, faint noise on very smooth surfaces even with TV loss, and the restriction of the current prototype to a single RGB sphere image rather than more complex geometries or SVBRDF settings (Jin et al., 24 Aug 2025). SaaFormer’s framing implies a different limitation landscape: its contribution is motivated by the observation that pixel-wise random sampling can induce data leakage via overlapping neighborhoods, so benchmark results under random partitions may not accurately reflect out-of-region generalization (Zhao et al., 2023).

Across both works, the broader significance of SSTA-like designs lies in explicit factorization of spectral and spatial or angular structure. In SpecGen this factorization enables reuse of abundant RGB BRDF data for spectral BRDF generation. In SaaFormer it suggests that disentangling spectral continuity from horizontal and vertical spatial aggregation can improve generalization under non-overlapping dataset partitions.