---
title: Subspace Approximation with Adjusted Bias (Saab)
url: https://www.emergentmind.com/topics/subspace-approximation-with-adjusted-bias-saab
type: topic
---

# Subspace Approximation with Adjusted Bias (Saab)

Subspace Approximation with Adjusted Bias (Saab) is a data-driven transform that augments PCA-style subspace projection with an explicit bias so that projected responses remain nonnegative after projection, thereby addressing the interaction between zero-mean projections and ReLU-style nonlinearities while retaining an interpretable DC/AC decomposition. In the cited literature, Saab appears as a feedforward alternative to backpropagation-designed convolutional layers, as an explainable orthonormal transform for intra video coding, and as a backpropagation-free feature extractor for medical-image segmentation; across these settings, its parameters are obtained from sample statistics such as means, covariances, eigenvectors, and extreme projection values rather than gradient descent [1810.02786][2012.11152][2107.10718][2607.02209].

## 1. Origins and problem setting

Saab was introduced in the context of interpretable feedforward design for convolutional neural networks, where direct use of PCA filters creates two stated difficulties: sign confusion and the destructive effect of ReLU on negative projections. In that setting, PCA filters produce outputs $y_k = a_k^T x$, and ReLU clips all negative responses to zero; standard PCA also has no built-in bias term to lift negative projections into the positive regime. Saab addresses this by augmenting PCA with a constant bias so that all channel outputs are nonnegative and by decoupling DC and AC subspaces so that the bias remains in the DC subspace of the output space [1810.02786].

A second formulation arises in transform coding, where transform design is posed as an optimization problem over coding-relevant criteria such as energy compaction and decorrelation. There, Saab is described as an explainable machine learning based transform and compared directly with the mainstream Discrete Cosine Transform (DCT) and with the Karhunen–Loève Transform (KLT). The transform is designed off-line from training data and then deployed as a fixed orthonormal transform without per-block adaptation overhead [2012.11152].

Later work embeds Saab in successive subspace learning and in multi-scale, wavelet-coupled pipelines for medical segmentation. In those systems, the same core idea—a covariance-derived orthonormal projection plus an analytically specified bias—is used as a front end for feature extraction, with supervised selectors and classical classifiers handling the final prediction stage [2107.10718][2607.02209].

## 2. Mathematical structure

In the feedforward-CNN formulation, Saab begins with an orthogonal decomposition of the input space into DC and AC components. For $x \in \mathbb{R}^N$,
$$
a_0 = \frac{1}{\sqrt{N}}[1,\dots,1]^T,\qquad
x_{\mathrm{DC}} = a_0^T x,\qquad
x_{\mathrm{AC}} = x - a_0 a_0^T x.
$$
The AC anchors are the top eigenvectors of the covariance of $x_{\mathrm{AC}}$, and the transform output is written
$$
y = W^T x + b,
$$
where the columns of $W$ are orthonormal anchors and the bias vector $b$ is chosen so that ReLU becomes the identity on the transformed outputs [1810.02786].

In the transform-coding formulation, the one-stage Saab transform for a $K$-dimensional vector $x$ is written
$$
y_0 = \frac{1}{\sqrt{K}}\,1^T x,\qquad
z' = x - y_0\,1,\qquad
y_k = a_k^T z' + b_k,\quad k=1,\dots,K-1.
$$
Here $y_0$ is the DC coefficient, the AC kernels $\{a_k\}$ are orthonormal eigenvectors of the covariance of $\{z'\}$, and the inverse takes the form $x = A_{\mathrm{Saab}}^T(y-b)$ because of orthogonality [2012.11152].

In MedSaab-US, Saab is specialized to wavelet subbands and patch-based local statistics. For each subband index $k$ and patch size $p \in \{5,11,21\}$, overlapping $p \times p$ patches $x \in \mathbb{R}^{p^2}$ are sampled, mean-adjusted by
$$
\mu_{k,p} = \frac{1}{M}\sum_{i=1}^M x_i,\qquad \tilde{x} = x - \mu_{k,p},
$$
and used to form the covariance
$$
C_{k,p} = \frac{1}{M}\sum_{i=1}^M \tilde{x}_i \tilde{x}_i^T.
$$
The top $K=8$ eigenvectors are retained, yielding the feature map
$$
F_j = u_j^T(x-\mu_{k,p}) + b_j,\quad j=1,\dots,K,\qquad
F = U^T(x-\mu_{k,p}) + b.
$$
Taken together, these formulations suggest that Saab is characterized operationally by two invariants: covariance-eigenvector subspace approximation and explicit bias adjustment [2607.02209].

## 3. Adjusted bias as the defining departure from standard PCA

The bias term is the central distinction between Saab and ordinary PCA. Standard PCA uses $F = U^T(x-\mu)$ and produces zero-mean outputs that can be positive or negative. Saab adds a constant offset to each projected dimension so that the outputs are nonnegative and the DC channel can be handled separately; in the feedforward interpretation, this makes the ReLU stage non-destructive, and in MedSaab-US it is described as enabling a built-in ReLU without information collapse [1810.02786][2607.02209].

The literature does not present a single universal bias rule. In the original feedforward formulation, the constraints are that $y_k = a_k^T x + b_k \ge 0$ for all $x$ and all $k$, and that the bias vector lie in the DC subspace of the output space; a sufficient bound is
$$
b_k \ge \max_x \|x\|,\quad k=0,\dots,K-1,
$$
with the practical choice $b_k = \max_{x \in \text{training}}\|x\| + \delta$ for small $\delta > 0$. In the cine-MRI segmentation formulation, a common choice is
$$
b_c = d\sqrt{F},\quad d>0,
$$
with $d$ selected large enough that every training projection remains nonnegative. In transform coding, the practical rule is $b_k = \max_{d \in D}\|d\|$ over the training set. In MedSaab-US, the bias is determined analytically on a per-channel basis by
$$
b_j = - \min_{i=1,\dots,M}(u_j^T \tilde{x}_i),
$$
which guarantees nonnegativity on the sampled training patches [2107.10718][2012.11152].

A common misconception is to equate Saab with ordinary PCA plus a post hoc shift. The cited formulations indicate a stronger claim: the bias is built into the transform design so that sign ambiguity, DC/AC handling, and subsequent stage composition are addressed at the representation level rather than by a separately learned nonlinearity [1810.02786][2012.11152].

## 4. Architectural realizations

In interpretable feedforward CNN design, multiple Saab transforms are cascaded to yield multiple convolutional layers, while fully connected layers are constructed using a cascade of multi-stage linear least squared regressors. The one-pass learning procedure computes the DC anchor, removes the DC component, estimates the AC covariance, extracts the top eigenvectors, selects the bias from training-set norms, and then applies convolution and identity ReLU at each spatial location. The reported architectures include LeNet-5 and a color-image variant with first convolution $5 \times 5 \times 3 \rightarrow 32$, second convolution $5 \times 5 \times 32 \rightarrow 64$, and fully connected layers $200 \rightarrow 100 \rightarrow 10$ [1810.02786].

In intra video coding, Saab is used as an off-line learned transform for $8 \times 8$ residual blocks. Residuals are collected per intra mode, grouped by fine-grained or coarse-grained grouping, and used to learn 24 distinct Saab transforms $\{SBT_k\}_{k=0}^{23}$. Three encoder integration strategies are defined. In $s_I$, selected modes use prelearned SBT kernels and the rest use DCT, with no per-TU flag. In $s_{II}$, DCT is active everywhere, but for modes with available SBT kernels the encoder performs rate–distortion comparison and signals the choice with a 1-bit flag per TU. In $s_{III}$, DCT plus all SBT kernels are activated, and every TU and mode chooses between DCT and the corresponding SBT via rate–distortion optimization [2012.11152].

In cine MRI segmentation, Saab is embedded in a successive subspace learning pipeline comprising sequential expansion of near-to-far neighborhood at different resolutions, channel-wise subspace approximation using Saab transform for unsupervised dimension reduction, class-wise entropy guided feature selection for supervised dimension reduction, concatenation of features and pixel-wise classification with gradient boost, and conditional random field for post-processing. The architecture uses $I=4$ Saab-based SSL units with interleaved max-pooling and empirically chosen anchors
$$
F_1=5,\;F_2=10,\;F_3=30,\;F_4=100,
$$
with neighborhood size fixed at $3 \times 3$ in all stages [2107.10718].

In thyroid ultrasound segmentation, MedSaab-US combines a 2-level Daubechies-4 DWT with multi-scale channel-wise Saab transforms at patch sizes $5 \times 5$, $11 \times 11$, and $21 \times 21$ on seven subbands. For each $(k,p)$ pair, $K=8$ output channels are kept, so the system produces $7 \times 3 \times 8 = 168$ Saab features per pixel. These are concatenated, augmented with three positional features $[r/H, c/W, \text{distance-to-center}]$, reduced by Label-Assisted Greedy (LAG) selection from 171 dimensions to the top 60 features, and then classified pixel-wise by an XGBoost ensemble with 600 trees, max depth 7, and learning rate 0.05. The framework is described as grounded in the Green Learning paradigm, and all Saab parameters are determined in closed form from $M=40\,000$ patches drawn from 400 training images [2607.02209].

## 5. Empirical record across application domains

The empirical literature on Saab is heterogeneous, spanning classification, transform coding, and medical segmentation. The reported results therefore quantify task-specific behavior rather than a single unified benchmark [1810.02786][2012.11152][2107.10718][2607.02209].

| Domain | Saab configuration | Reported outcome |
|---|---|---|
| Feedforward CNN classification | Saab conv layers, FF and Hybrid variants | MNIST: FF 97.2%, Hybrid 98.4%, BP 99.9%; CIFAR-10: FF 62%, Hybrid 64%, BP 68% |
| HEVC intra coding | 8$\times$8 Saab transform with $s_I$, $s_{II}$, $s_{III}$ | Avg. BDBR = -1.41%, -2.59%, -3.07%; max -10.00% |
| Cine MRI segmentation | SSL + Saab | With 50 training subjects: Avg Dice 85.12 vs U-Net 83.35 and AttenU-Net 82.91; with 60 training subjects: Avg Dice = 86.03% |
| Thyroid ultrasound segmentation | MedSaab-US | Mean Dice coefficient 0.4784 +/- 0.2190, precision 0.5768, recall 0.5604; model footprint under 500K parameters; CPU-only inference in approximately 0.3 seconds per image |

Additional quantitative findings refine this picture. In transform coding, the decorrelation cost averaged over several sequences and QPs is reported as KLT: 0, Saab$(8 \times 8)$: $\approx 2345$, Saab$(4 \times 4 \rightarrow 2 \times 2)$: $\approx 2442$, and DCT: $\approx 2579$; at 90% cumulative energy, one-stage Saab typically outperforms DCT by 5–10%. In the same study, under the full rate–distortion selection strategy, on average 46.1% of TUs choose Saab over DCT, with peaks at $>80\%$ for textured sequences [2012.11152].

In the feedforward-CNN study, adversarial robustness is also reported. Against attacks targeting the BP-trained net, BP accuracy drops to 6%/1%/2% on MNIST and 15%/12%/15% on CIFAR-10 under FGS, BIM, and DeepFool, whereas the FF design retains 56%/46%/96% on MNIST and 21%/31%/59% on CIFAR-10. The same study states that attacks targeting the FF-designed net leave FF suffering less under high-quality attacks than BP [1810.02786].

## 6. Interpretability, efficiency, and limitations

Saab is consistently positioned as an interpretable alternative to backpropagation-based filter learning. In the feedforward formulation, the filters are data-driven PCA patterns, including Gabor-like edge detectors in early layers and compound contours in deeper layers; orthogonality yields uncorrelated feature channels, and the explicit bias removal of ReLU nonlinearity is part of the analytic construction. In medical segmentation, the same property is used to justify a lightweight and interpretable machine learning model; in transform coding, it is framed as explainable machine learning based transform design [1810.02786][2107.10718][2012.11152].

Its computational profile depends strongly on the deployment domain. In MedSaab-US, each Saab transform at scale $p$ has $p^2 \times K$ float weights for $U$ plus $K$ bias scalars, giving an overall Saab parameter count of approximately 33,080 parameters, which is described as negligible compared to the approximately 500 K total pipeline parameters because most parameters reside in the XGBoost ensemble. The corresponding computational cost per pixel is
$$
O(7 \times (25+121+441)\times 8) \approx 7 \times 587 \times 8 \approx 32\,888
$$
multiply–adds per pixel, and for a $256 \times 256$ image this is approximately $2.1 \times 10^9$ operations, reported to run in approximately $0.3$ s on a 2 GHz dual-core CPU [2607.02209].

The medical-segmentation studies emphasize parameter efficiency. The cine-MRI SSL + Saab model has 0.03 M parameters and is reported as having approximately 200$\times$ fewer parameters than U-Net. MedSaab-US reports a model footprint under 500K parameters and CPU-only inference. These are contrasted with deep learning systems that use millions of parameters and GPU-dependent training via backpropagation [2107.10718][2607.02209].

The limitations are equally explicit. In the feedforward-CNN study, FF performance lags BP on both MNIST and CIFAR-10, and the discussion notes that further work is needed on extremely deep nets and large-scale tasks; bias selection also requires norm bounds on inputs. In video coding, Saab improves coding efficiency but incurs nontrivial computational overhead relative to optimized integer-DCT, with encoding time ratios of 202%, 240%, and 284% and decoding time ratios of 148%, 134%, and 138% for $s_I$, $s_{II}$, and $s_{III}$, respectively. In thyroid ultrasound segmentation, MedSaab-US is explicitly presented as an exploratory non-DL baseline and includes analysis of the specific challenges posed by isoechoic nodules [1810.02786][2012.11152][2607.02209].

A plausible implication is that Saab is best understood not as a drop-in replacement for every backpropagation-based module, but as a mathematically tractable design pattern for situations where closed-form parameter determination, interpretability, and modest model size are primary constraints.

Source: https://www.emergentmind.com/topics/subspace-approximation-with-adjusted-bias-saab