mSDA: Marginalized Stacked Denoising Autoencoders

• mSDA is a scalable, closed-form unsupervised learning model that marginalizes input noise to extract robust features without iterative optimization.
• It uses linear algebra techniques to compute denoising transforms quickly, achieving dramatic speedups in high-dimensional data scenarios.
• mSDA excels in domain adaptation tasks by efficiently handling sparse, noise-prone text data, outperforming traditional gradient-based SDAs in speed and performance.

Marginalized Stacked Denoising Autoencoders (mSDA) are a class of deep unsupervised feature learning models developed to address the computational inefficiencies and scalability limitations inherent in traditional Stacked Denoising Autoencoders (SDA). mSDA introduces a closed-form, convex training procedure that marginalizes input corruption noise analytically, leading to substantial gains in training speed and reproducibility while retaining or surpassing the representational power of conventional SDAs. The method is particularly suited to high-dimensional data, notably bag-of-words representations in text-based domain adaptation tasks (Xu et al., 2011, Chen et al., 2012).

1. Motivation and Comparative Background

Standard SDAs learn data representations by reconstructing input vectors with an encoder-decoder architecture trained on stochastically corrupted data. While successful in improving the accuracy of shallow classifiers (e.g., SVMs), especially in domain adaptation, conventional SDAs exhibit major drawbacks for large-scale or high-dimensional problems:

• Training requires iterative optimization (stochastic gradient descent), which is slow—especially for large input space (e.g., d in the order of $10^4$ or higher) and sample size $n$.
• Numerous hyperparameters (learning rate, batch size, hidden dimensions, number of epochs, corruption probability) must be tuned, rendering training time-consuming and unstable due to the objective’s non-convexity.
• Scalability is poor for high-dimensional data prevalent in natural language processing.

mSDA replaces iterative optimization with a procedure that marginalizes the corruption process and computes representations in closed form using linear algebra, yielding large theoretical and empirical speedups. It is straightforward to implement, requiring fewer than 20 lines of MATLAB code (Xu et al., 2011, Chen et al., 2012).

2. Denoising Objective and Marginalization of Corruption

The foundational objective of mSDA is to learn a linear transformation $W$ that reconstructs a clean input $x$ from its corrupted version $\tilde{x}$. The corruption is applied independently to each input feature with masking noise—setting coordinates to zero with probability $p$, otherwise leaving them unchanged.

Given $X = [x_1, ..., x_n] \in \mathbb{R}^{d \times n}$ (optionally augmented by a bias), the corrupted input is defined as $\tilde{x} = \mathring{\nu}\, x$, where $\mathring{\nu}$ is a diagonal Bernoulli matrix.

The loss function is the expected squared reconstruction loss (with ridge regularization):

$$L(W) = \mathbb{E}_{x}\, \mathbb{E}_{\mathring{\nu}} \left[ \|x - W\tilde{x} \|_2^2 \right] + \lambda \|W\|_F^2$$

Rather than empirically averaging over a finite number of corruptions or optimizing with SGD, mSDA analytically computes the expectations involved, producing a marginalized loss.

3. Closed-Form Solution and Layer-Wise Construction

Single-layer computation

For the data matrix $n$0:

• Observed (uncorrupted) covariance: $n$1
• Expectation of corrupted–corrupted covariance:

$n$2

• Expectation of clean–corrupted cross-covariance:

$n$3

• Closed-form mapping (with regularization $n$4):

$n$5

No iterative optimization is needed; the solution requires only matrix inversions of size $n$6.

Stacking to form mSDA

To increase representational capacity, several layers of linear denoisers are composed. Denote $n$7 and for each layer $n$8:

• $n$9
• Form $W$0, $W$1 as above for $W$2
• Compute $W$3
• Compute $W$4
• Apply pointwise nonlinearity $W$5 (e.g., thresholding or $W$6): $W$7

The final feature set for downstream tasks is the concatenation $W$8.

4. Complexity, Scalability, and Implementation Details

Each mSDA layer requires $W$9 time for covariance computation and $x$0 time for solving a $x$1 linear system. Memory usage scales as $x$2 for storing covariance matrices. For extremely high-dimensional problems (e.g., $x$3), a sub-block or "pivot feature" strategy can reduce cost by partitioning features and learning smaller mDA mappings on feature subsets before stacking (Chen et al., 2012).

Key tips:

• Use a small ridge $x$4 (e.g., $x$5 to $x$6) to regularize matrix inversion.
• Typical noise probabilities are $x$7.
• Three to five layers typically suffice; deeper stacking often yields diminishing returns.
• Feature normalization (e.g., $x$8 normalization column-wise) before downstream classification may be beneficial.

5. Empirical Evaluation and Benchmark Results

On sentiment analysis domain adaptation benchmarks using Amazon product review data, mSDA matches or slightly outperforms traditional SDA in 10 out of 12 transfer tasks with markedly lower computational requirements:

Method	1-layer Training Time	5-layer Training Time	Typical Accuracy
SDA (SGD-trained)	5 hours	2 days	Baseline
mSDA (closed form)	25 seconds	2 minutes	Equal or greater

• mSDA yields gains of 1–3% over raw bag-of-words or PCA features in standard benchmarks.
• For $x$9, $\tilde{x}$0, a three-layer mSDA matched SDA's error in 14 minutes, a $\tilde{x}$1 speedup.
• Increasing $\tilde{x}$2 (including rare features) improved transfer ratio by up to 5%, with mSDA maintaining competitive accuracy and speed (Chen et al., 2012, Xu et al., 2011).

6. Domain Adaptation Applications

mSDA is especially effective in unsupervised domain transfer tasks, where it is used to generate data representations that bridge the gap between disparate domains by training jointly on (unlabeled) source and target data. On Amazon sentiment datasets, combining source and target domain features then passing them through stacked mSDA yielded transfer accuracies and error ratios at par with or better than previous state-of-the-art methods, including non-linear SDA and SCL/CODA baselines (Chen et al., 2012).

mSDA has been shown to provide robust representations for high-dimensional text by marginalizing the effects of missing or rare features and facilitating information sharing across feature subsets in deeper layers.

7. Practical Recommendations and Limitations

• Corruption rate $\tilde{x}$3 should be neither too low (trivial identity mapping) nor too high (excessive information loss); $\tilde{x}$4 is typically effective for text.
• For numerically stable inversion when $\tilde{x}$5 is large, prefer Cholesky over explicit matrix inversion.
• For extremely high $\tilde{x}$6, construct the first-layer mapping on blocks of $\tilde{x}$7 pivot features, then stack as usual.
• Empirical results suggest performance saturates after 3-5 layers; adding more layers yields minimal benefit.
• Implementation requires storing only a handful of $\tilde{x}$8 matrices and does not require hyperparameter tuning related to learning rates or momentum.

mSDA presents a scalable, fast, and effective alternative to gradient-based unsupervised representation learning for domain adaptation and other tasks involving high-dimensional sparse data. Its key innovation lies in the analytical marginalization of corruption noise and stackable closed-form denoising transforms, which collectively enable both rapid computation and strong empirical performance (Xu et al., 2011, Chen et al., 2012).