Layer-wise Linear Decomposition

• Layer-wise linear decomposition is a method for factorizing, approximating, and interpreting the linear operations in each deep neural network layer to enhance model efficiency and explainability.
• It utilizes techniques such as layer-wise relevance propagation, matrix/tensor decompositions, and rotor-based approaches to reveal latent structures within networks.
• The approach underpins applications in model compression and efficient parameterization, with empirical results validating its practical benefits on benchmark architectures.

Layer-wise linear decomposition refers to a collection of analytical and algorithmic frameworks for factoring, approximating, or interpreting the linear (or linearizable) operations applied in each layer of deep neural networks. This family includes techniques for explainability (layer-wise relevance propagation), compression (structured low-rank or tensor decompositions), efficient parameterization (e.g., rotor/gadget-based or factorized layers), and geometric function analysis (latent manifold interpretation). By exposing explicit or approximate linear structure on a per-layer basis, layer-wise linear decomposition supports interpretability, resource-efficient deployment, and mathematical understanding of network behavior.

1. Foundational Layer-wise Linear Decomposition Methods

Central techniques for layer-wise linear decomposition are grounded in either analytic expansion—explicitly expressing nonlinear network outputs as combinations of local linear functions—or algebraic/spectral decompositions of layer weight matrices. Prominent approaches include:

• Layer-wise Relevance Propagation (LRP): Computes local heatmaps that assign input-level relevance via linear redistribution rules applied at each layer. For deep ReLU networks, the propagation relies on the α/β-rule, which splits positive and negative contributions, and, for models with unstable denominators (e.g., Fisher Vector pipelines), on ε-stabilized division. The depth of this decomposition is tunable, enabling fine control over the semantic and spatial granularity of explanations (Bach et al., 2016, Kobayashi et al., 2023).
• Matrix and Tensor Decompositions: Weight matrices in linear or convolutional layers are subject to SVD or tensor methods such as tensor-train, Tucker, or grouped low-rank factorization. This enables approximating layers with significantly reduced rank or parameterization, which supports efficient inference and model compression (Liebenwein et al., 2021, Gray et al., 2019). The Eckart–Young–Mirsky theorem formally characterizes the optimal error of rank-k approximations.
• Spectral and Weight-Based Tensor Decomposition (Bilinear MLPs): Bilinear gating layers can be exactly decomposed into eigenvector expansions at the layer level, yielding sparsely interacting features that directly reflect model weights. This supports interpretability and enables architecture-level modifications for built-in analysis (Pearce et al., 2024).
• Geometric and Clifford-Algebraic Decomposition: Arbitrary linear operators can be constructed as products of geometric primitives—specifically, rotors in Clifford algebra. This “rotor decomposition” provides a highly parameter-efficient layer representation with performance competitive to low-rank and block-Hadamard baselines, specifically in attention projections (Pence et al., 15 Jul 2025).

2. Theoretical Principles and Error Guarantees

Layer-wise linear decomposition frameworks rely on rigorous theoretical results:

• Eckart–Young–Mirsky Theorem: For any matrix $W$, the optimal rank-$r$ approximation (in the Frobenius or operator norm) is given by truncating to its top-$r$ singular values. For a group-sliced or fully unfolded layer tensor, this sets the minimal achievable reconstruction error for a given compression (Liebenwein et al., 2021, Shyh-Chang et al., 2023).
• Encoding Theorem for Neural Networks: For any stably converged neural network with continuous activation functions, each layer’s weight matrix encodes a (truncated) continuous function approximating its local data manifold. The error due to truncation (e.g., SVD) at each layer sums telescopically to bound the total network function approximation error (Shyh-Chang et al., 2023).
• Compression-Discrimination Law: In deep linear networks, feature compression (within-class variance) decays geometrically through layers, while discrimination (between-class separation) increases approximately linearly. This layer-wise analysis predicts empirical trends in both linear and nonlinear, pretrained networks (Wang et al., 2023).
• Clifford-Algebraic Universality: Any real $d\times d$ linear transformation can be synthesized (up to machine precision) from $O(\log^2 d)$ parameters using rotor sandwiches in Clifford algebra, establishing strong parameter efficiency without loss of representational power (Pence et al., 15 Jul 2025).

3. Practical Decomposition Algorithms and Optimization

Efficient implementation and practical deployment require specialized algorithms:

• Depth-Controlled LRP: LRP can be implemented in a single forward-backward pass. The decomposition “cut-off point” $l_\mathrm{cut}$ determines whether detailed LRP rules or uniform redistribution is applied below a given layer. This enables a flexible explanatory resolution knob and efficient, context-aware computation (Bach et al., 2016, Kobayashi et al., 2023).
• ALDS (Alternating Layer Decomposition Search): Solves for the globally optimal set of per-layer SVD ranks focusing on parsimony (parameter/FLOP budget) and accuracy. An EM-style procedure alternates between global allocation and local refinement steps, using precomputed error lookup tables for efficiency (Liebenwein et al., 2021).
• Structured Low-Rank and Factorized Substitutions: Layer-wise replacements (ACDC, Tensor-Train, Tucker, HashedNet, ShuffleNet-linear, linear bottleneck) are trained end-to-end via standard SGD, often with specialized initialization and parameterized regularization such as compression-ratio-scaled (CRS) weight decay. Empirical results favor attention-transfer or knowledge distillation to regain performance in compressed models (Gray et al., 2019).
• Differentiable Geometric Decomposition: Rotors are constructed by exponentiating bivectors, with invariant decomposition into simple components using power-iteration. This process is fully differentiable and can be integrated into modern deep learning frameworks. Parameter counting and closed-form expressions ensure practical deployment for large attention layers (Pence et al., 15 Jul 2025).
• Weight-Based Spectral Decomposition for Bilinear Layers: The bilinear tensor is reduced into symmetric matrices for each output direction, eigendecomposed, and optionally sparsified via eigenvalue truncation. Forward and backward passes are exactly reconstructed, preserving model gradients and semantics (Pearce et al., 2024).

4. Interpretability, Resolution Control, and Semantic Granularity

Layer-wise linear decomposition enables a fine trade-off between interpretability, computational efficiency, and semantic detail:

• Resolution versus Semantics: In LRP, deep propagation (low cut-off index) yields high-resolution heatmaps pinpointing pixel/subpixel contributions, while shallow propagation highlights broader semantic regions (object- or region-level cues) (Bach et al., 2016). For Fisher Vector or DNN pipelines, the limiting heatmap granularity tracks feature map spatial binning or convolutional filter sizes.
• Explainability Validation: Self-supervised corruption-recall protocols—where relevant input features are corrupted and explanation recall rates are measured—demonstrate that layer-wise linear decompositions such as LRP can recover ground-truth anomalies or feature salience substantially more efficiently and accurately than sampling-based methods (e.g., SHAP) (Kobayashi et al., 2023).
• Feature Evolution Across Layers: The compression-discrimination framework empirically shows that over-collapsed representations (excessive within-class compression) in deep layers can impede transfer; optimal feature reuse—e.g., via projection heads—aligns with explaining prediction coverage at intermediate layers (Wang et al., 2023).
• Bilinear Spectral Interpretability: Bilinear MLP eigenvectors correspond to interpretable patterns (digit strokes, semantic feature directions), and the sorted eigenvalue spectrum allows direct ranking and pruning of features. Late layers tend to develop strong positive and negative eigen-features differentiating precise, sparse cues from more global, vague structures (Pearce et al., 2024).

5. Compression, Efficiency, and Parameterization

Layer-wise linear decompositions underpin state-of-the-art compression and efficiency strategies:

• Low-Rank and Groupwise SVD Compression: Slicing channels and applying per-group SVD greatly increases the attainable global compression ratio, enabling aggressive parameter/FLOP reduction with minimal accuracy loss (e.g., up to 75% parameter removal at <1% Top-1 accuracy drop in ResNet/ImageNet) (Liebenwein et al., 2021).
• Structured Substitutions in Convolutions: Pointwise convolutions (1×1) dominate parameter count in modern architectures. Substitution with structured transforms (ACDC, TT, HashNet, ShuffleNet-linear) provides Pareto-optimal control, delivering up to 10-100× savings with ~1%–3% accuracy cost. CRS weight decay and attention transfer are critical for stable optimization (Gray et al., 2019).
• Rotor and Clifford-Algebraic Layers: Rotor gadgets scale as $O(\log^2 d)$ parameters per layer, orders of magnitude smaller than dense or low-rank alternatives, yet match or exceed their accuracy on key tasks (e.g., when replacing transformer attention projections in LLaMA-1B) (Pence et al., 15 Jul 2025).
• Hybrid Linear-Softmax Attention: Layer-wise hybridization designs, as in SoLA-Vision, interleave efficient linear attention with sparse global softmax layers. Theoretical analysis reveals that stacking linear layers grows receptive field only as $\mathcal{O}(\sqrt{M})$, necessitating “softmax shortcuts” for full global modeling. Empirically, such layer-wise hybrids outperform both pure and traditional windowed hybrids for image classification and dense prediction (Li et al., 16 Jan 2026).

6. Empirical Results, Limitations, and Outlook

Empirical studies across tasks and architectures substantiate the practical value and interpretative insights of layer-wise linear decompositions:

• LRP and Decomposition Cut-Off: On PASCAL VOC 2007, deep CNNs achieve mAP = 72.1%, FV classifiers mAP = 60.0%. High-resolution, full-depth heatmaps yield more localized and specific features; shallow cut-off yields coarser semantic maps but facilitates region-level understanding (Bach et al., 2016).
• Groupwise Low-Rank Compression: ALDS achieves up to 95% parameter cut (VGG16/CIFAR-10) and 81% FLOPs removal (AlexNet/ImageNet) with ≤0.5–1% accuracy drop (Liebenwein et al., 2021).
• Rotor and Bilinear Gadgets: Rotor-based layers match or slightly outperform low-rank and block-Hadamard on language modeling and multiple-choice tasks, with extreme parametric efficiency (Pence et al., 15 Jul 2025). Eigen-decomposed bilinear MLPs achieve interpretable and compressible features, maintaining competitive val loss when used in TinyLlama-1.1B (Pearce et al., 2024).
• Limitations: Some decompositions (Tucker, TT) require complex kernel implementations at inference; HashedNet and similar techniques may not reduce FLOPs despite parameter reduction. Rotor gadgets’ computational efficiency is currently limited by dense realization; extension to fully trained-from-scratch settings is an open direction (Pence et al., 15 Jul 2025, Gray et al., 2019).
• Prospective Research: Dynamic, adaptive rank regularization; application of geometric decomposition for interpretability; robust topology- and spectrum-aware initialization; hybridization strategies for attention; and circuit-level intervention using interpretable subgraphs are active research avenues across the layer-wise decomposition landscape.

---

Selected References:

• "Controlling Explanatory Heatmap Resolution and Semantics via Decomposition Depth" (Bach et al., 2016)
• "Unlocking Layer-wise Relevance Propagation for Autoencoders" (Kobayashi et al., 2023)
• "Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition" (Liebenwein et al., 2021)
• "Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity" (Shyh-Chang et al., 2023)
• "Separable Layers Enable Structured Efficient Linear Substitutions" (Gray et al., 2019)
• "Composing Linear Layers from Irreducibles" (Pence et al., 15 Jul 2025)
• "Weight-based Decomposition: A Case for Bilinear MLPs" (Pearce et al., 2024)
• "Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination" (Wang et al., 2023)
• "SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention" (Li et al., 16 Jan 2026)