--- title: VarPro & Over-Parameterization url: https://www.emergentmind.com/topics/variable-projection-scheme-based-on-over-parameterization type: topic --- # VarPro & Over-Parameterization A variable projection scheme based on over-parameterization refers to an optimization methodology that exploits problem structure by partitioning parameters into blocks (typically linear and nonlinear or coupled and decoupled) and then marginalizing—i.e., analytically eliminating or solving for—one block given the other, usually enabled by smooth convexity and/or low-dimensionality of the “inner” block. Over-parameterization refers to the use of redundant or expanded parameterizations that provide flexibility, smoothness, or improved conditioning for the overall optimization process. This approach has seen rigorous development in DNN training, structured low-rank approximation, time series analysis, polynomial common divisor problems, tensor algebra, and non-smooth regularized regression. ## 1. Key Concepts and Structural Formulation Variable projection (VarPro) originated for separable nonlinear least squares, in which an objective $$ \min_{w, v}\quad \Phi(w, v) = \frac{1}{N}\sum_{i=1}^N L\big(V\,F(x_i; W), c_i\big) + R(W) + S(V) $$ is split such that $W$ parameterizes a nonlinear block (e.g., DNN feature extractor) and $V$ a linear block (e.g., last layer, linear weights) [2007.13171]. Over-parameterization arises by treating $W$ and $V$ as independent and then explicitly minimizing over $V$ for fixed $W$: $$ V(W) = \arg\min_V\,\Phi(W, V) $$ yielding a reduced objective $$ \Phi_{\mathrm{red}}(W) = \Phi(W, V(W)) $$ Optimization then proceeds in $W$ only, with $V$ kept in analytic equilibrium. This structure generalizes across domains (e.g., polynomial quotient/divisor form [1304.6962], tensor transformation/representation [2406.06942], low-rank signal subspaces [2101.09779], structured matrix approximation [1211.3938], non-smooth convex problems [2205.01385]). The over-parameterized aspect typically refers to expanding the parameter space so that analytic or semi-analytic elimination is feasible and the remaining variables enjoy better curvature or conditioning (e.g., eliminating vast numbers of linear coefficients while only iterating nonlinear weights). ## 2. Algorithmic Schemes and Computational Properties ### Gauss-Newton Variable Projection (GNvpro) In DNN training [2007.13171], the GNvpro method involves: - **Inner solve:** For least-squares loss, $V(W)$ has a closed-form solution via normal equations; for cross-entropy, a small Newton-Krylov trust-region solver solves for $V$. - **Outer solve:** Compute gradients, Jacobian-vector products, and Hessians (using the Gauss-Newton approximation) only with respect to $W$, exploiting the vanishing gradient property of $V$ due to optimality. - **Trust-region iteration:** Krylov subspace methods (Arnoldi/GMRES) optimize step $\delta$ in $W$ within trust-region constraints. This results in accelerated convergence (tens of passes, versus thousands for SGD), minimal cost for the inner block, and improved stability due to reduced coupling ill-conditioning. ### Structured SLRA and Polynomial GCD Structured low-rank approximations (affine mosaics, Hankel, Sylvester matrices) [1304.6962, 1211.3938] employ over-parameterized representations—partitioning parameters into linear and nonlinear/annihilator blocks and eliminating the linear part using least-squares/least-norm solutions, yielding reduced cost functions with closed-form gradients and efficient block-banded Cholesky solves. In approximate GCD computations, the image representation seeks polynomials $(p_1, ..., p_N) = (h g_1, ..., h g_N)$, eliminates $g_i$ for fixed $h$, and optimizes a reduced cost $\varphi(h)$ with guaranteed linear complexity for small/large degree regimes [1304.6962]. ### Time Series and Signal Subspaces Low-rank signal subspaces parameterized via Generalized Linear Recurrence Relations (GLRRs) exploit over-parameterization $(s, \theta)$ to represent constraints, with variable projection and FFT-regularized basis construction for stable, low-cost projection and gradient computation [2101.09779]. ### Non-Smooth Structured Regression Over-parameterized Hadamard (U–V) formulations rewrite non-smooth penalties (e.g., group-lasso, TV) into smooth, albeit nonconvex, quadratic forms. Variable projection eliminates one variable block, yielding a smooth objective for quasi-Newton optimization, with dimension-free convergence rates and no need for parameter tuning [2205.01385]. ## 3. Theoretical Properties: Convergence, Conditioning, and Uniqueness Variable projection with over-parameterization often improves the conditioning of the Hessian of the reduced objective by separating out ill-conditioned (typically linear) variables. Strict convexity or affine-linear structure in the eliminated block ensures that the reduced problem is well-posed [2007.13171, 1211.3938]. This results in a lower-dimensional and smoother effective nonlinear optimization. Convergence guarantees hold under mild regularity; e.g., in tensor frameworks, the Riemannian gradient method converges at $O(1/\varepsilon^2)$ [2406.06942]. For GLRR signal models, diffeomorphic local parameterization ensures existence of stable projections [2101.09779]. The structure of the reduced cost prevents “peaking” phenomena observed in naive over-parameterized regressions [2011.11477]. Where uniqueness is analyzable, e.g., in tensor transforms (optimal $t$-SVD decompositions), solutions are unique up to row permutations/sign flips [2406.06942]. ## 4. Applications Across Domains ### Table: Representative Applications | Domain | Variable Partition | Projection Task | |------------------------------|---------------------------|---------------------------------------------------| | Deep Neural Networks [2007.13171] | (W: nonlinear, V: linear) | Optimize feature extractor, eliminate affine head | | Structured Polynomials [1304.6962] | (h: divisor, g: quotient) | Approximate GCD via analytic elimination | | Low-Rank Approximation [1211.3938] | (R: annihilator, p: parameters)| Structured low-rank matrix projection | | Signal Subspace [2101.09779] | (s, θ: GLRR parameters) | Time-series denoising, energy subspace modeling | | Tensor Decomposition [2406.06942] | (M: transform, X: tensor) | Learn matrix-mimetic tensor algebras | | Non-smooth Regression [2205.01385] | (u: inner, v: scaling) | Group Lasso, TV, square-root Lasso, ℓ_q recovery | Surrogate modeling with PDEs, hyperspectral segmentation, and image classification tasks benefit from GNvpro’s efficiency and superior generalization relative to stochastic gradient descent [2007.13171]. PCA-OLS and variant projection estimators in high-dimensional regression regularize and yield robust estimators, avoiding adversarial susceptibility in the overparameterized regime [2011.11477]. Matrix-mimetic tensor regression and compression achieve optimal transforms and representations, outperforming fixed heuristics such as DCT and vanilla SVD, with direct interpretability and generalization [2406.06942]. ## 5. Computational Complexity and Implementation Strategies The central computational advantage is the potential to marginalize the (typically) linear, convex block in closed form or via fast solvers, resulting in cost savings that scale linearly or near-linearly in problem size. For deep nets, the inner affine block is small compared to nonlinear layers, making its elimination cheap regardless of depth/width [2007.13171]. Structured low-rank mosaic Hankel or block-banded matrices allow efficient Cholesky or FFT-based projections [1211.3938, 2101.09779]. Outer nonlinear variable optimization is now only exposed to the smooth reduced objective, tractable by trust-region, conjugate gradient, L-BFGS, or Riemannian gradient descent (for orthogonal manifolds). Parameter selection (e.g., number of components $k$ in PCA-OLS) becomes a smooth, variable selection/projection task, with cross-validation or spectral-gap heuristics guiding dimension choice [2011.11477]. ## 6. Limitations and Open Considerations Over-parameterization via variable projection relies on strict convexity/smoothness of the inner block; e.g., least-squares and cross-entropy with $\ell_2$ regularization satisfy this, while combinations such as ReLU + $\ell_1$ require modification [2007.13171]. Matrix conditioning—absence of roots on the unit circle—is essential for efficient projection, particularly in FFT-based schemes [1304.6962, 2101.09779]. A plausible implication is that while over-parameterization generally accelerates optimization by decoupling and smoothing, problem-specific algebraic singularities or poorly conditioned underlying linear systems may limit the effectiveness of this strategy. ## 7. Impact and Future Directions Variable projection schemes grounded in over-parameterization have reshaped the effective practice and theory of high-dimensional function approximation, signal estimation, tensor decomposition, and structured regression. By enabling analytic elimination of vast blocks of parameters, these approaches deliver improved convergence, robustness, statistical generalization, and direct interpretability. Their application in matrix-mimetic tensor methods and DNN optimization foreshadows further integration into scalable machine learning and computational mathematics workflows—particularly as new structured models and problem representations emerge requiring flexible, smooth, and efficiently marginalizable parameterizations. Empirical and theoretical advances across the referenced domains suggest continuing expansion and refinement, with further exploration warranted in nonconvex and non-smooth settings, manifold-constrained optimization, and broader regularized inverse problem classes. Source: https://www.emergentmind.com/topics/variable-projection-scheme-based-on-over-parameterization