Sparse Variable Projection in Robotic Perception: Exploiting Separable Structure for Efficient Nonlinear Optimization

Abstract: Robotic perception often requires solving large nonlinear least-squares (NLS) problems. While sparsity has been well-exploited to scale solvers, a complementary and underexploited structure is \emph{separability} -- where some variables (e.g., visual landmarks) appear linearly in the residuals and, for any estimate of the remaining variables (e.g., poses), have a closed-form solution. Variable projection (VarPro) methods are a family of techniques that exploit this structure by analytically eliminating the linear variables and presenting a reduced problem in the remaining variables that has favorable properties. However, VarPro has seen limited use in robotic perception; a major challenge arises from gauge symmetries (e.g., cost invariance to global shifts and rotations), which are common in perception and induce specific computational challenges in standard VarPro approaches. We present a VarPro scheme designed for problems with gauge symmetries that jointly exploits separability and sparsity. Our method can be applied as a one-time preprocessing step to construct a \emph{matrix-free Schur complement operator}. This operator allows efficient evaluation of costs, gradients, and Hessian-vector products of the reduced problem and readily integrates with standard iterative NLS solvers. We provide precise conditions under which our method applies, and describe extensions when these conditions are only partially met. Across synthetic and real benchmarks in SLAM, SNL, and SfM, our approach achieves up to \textbf{2$\times$--35$\times$ faster runtimes} than state-of-the-art methods while maintaining accuracy. We release an open-source C++ implementation and all datasets from our experiments.

• The paper introduces a matrix-free VarPro framework that analytically eliminates linear variables while preserving problem sparsity in robotic perception.
• It achieves a 2–35× runtime reduction and improved memory efficiency by leveraging sparse Cholesky factorizations and a matrix-free Schur complement operator.
• The study outlines necessary conditions for effective variable elimination and offers guidance for future extensions in SLAM, SfM, and sensor network localization.

Sparse Variable Projection in Robotic Perception: Exploiting Separable Structure for Efficient Nonlinear Optimization

Introduction and Motivation

This paper targets the acceleration of large-scale nonlinear least-squares (NLS) optimization problems arising in robotic perception tasks such as SLAM, structure from motion (SfM), and sensor network localization (SNL), which routinely involve $10^5\!-\!10^6$ variables and display considerable sparsity and separability in their formulation. While sparsity-exploiting techniques have become standard for scalability, the separable structure—where linearity in some variables permits their analytical elimination—has been comparatively underleveraged, largely due to computational challenges introduced by gauge symmetries and rank deficiencies.

The central contribution is a matrix-free variable projection (VarPro) framework, implemented as a one-time preprocessing step, that eliminates unconstrained linear variables while preserving the original problem's sparsity pattern. This is achieved through an efficient computation of the Schur complement operator, facilitating fast cost, gradient, and Hessian-vector product evaluation for integration with standard iterative solvers.

Figure 1: (Top) Overview of variable projection reducing problem size without losing original sparsity; (Bottom) Cost–runtime comparison on BAL-1934 illustrating substantial performance gains after variable projection.

Problem Formulation and Theoretical Contributions

The paper considers a generalized matrix-valued NLS problem:

$$\min_{X_c \in \mathcal{X}, X_f} \sum_i \|A_i \begin{bmatrix} X_c \ X_f \end{bmatrix}\|^2_{\Omega_i},$$

where $X_c$ are typically robot poses (constrained variables), $X_f$ are landmarks or similar structures (unconstrained and appear linearly), and $A_i$ represents the measurement Jacobians. Under the common assumption of homogeneous linear residuals, the total cost assumes a quadratic form, permitting a block partitioning of the cost matrix $Q$ and variable elimination via the Schur complement.

A major theoretical advance is the provision of necessary and sufficient conditions for the applicability of this VarPro scheme, notably: (i) the existence of a subset of unconstrained variables, (ii) linearity of residuals in these variables, and (iii) a specific graph-theoretic structure in the residual Jacobian. The paper further analyzes cases of partial satisfaction of these conditions.

Critically, the key computational bottleneck—formation of the (possibly dense) Schur complement via the Moore-Penrose pseudoinverse—is circumvented with a matrix-free approach, leveraging CR decomposition and sparse Cholesky factorizations. For problems whose residuals can be interpreted as incidence matrices (common in SLAM/SfM), the Schur complement is expressed via reduced Laplacians, maintaining sparsity.

Practical Algorithm and Implementation

The approach constructs, in preprocessing, a matrix-free Schur complement operator by:

1. Identifying a sparse column basis $C$ for $A_f$ (the block Jacobian with respect to $X_f$), via either direct manipulation or rank-revealing QR.
2. Computing and storing a sparse Cholesky factorization $L$ of $C^\top \Omega C$ (the reduced Laplacian).
3. For each matrix-vector product during optimization, applying triangular solves with $L$ and multiplications with $B = \Omega C$.

This pipeline respects the original problem's sparsity, sidesteps dense matrix formation, and integrates into any iterative quadratic solver requiring only matrix-vector products with the Schur complement.

Figure 3: Comparative convergence (cost vs. iterations and cost vs. time) for various methods on representative PGO, RA-SLAM, SNL, and SfM problems; ours consistently achieves superior wall time due to lowered per-iteration cost, albeit direct solvers may require fewer iterations.

Experimental Evaluation

A comprehensive suite of experiments spans synthetic and real SLAM, SNL, and SfM datasets. The evaluation compares the proposed method, a naïve variable projection within the original system, and GTSAM (a state-of-the-art direct-factorization-based solver). Results are consistently in favor of the proposed approach:

• Runtime Reduction: Achieves 2–35× speedup over baselines on large-scale datasets, especially pronounced when the unconstrained variable set dominates (e.g., SfM's landmarks).
• Memory Efficiency: Successfully processes problem instances where GTSAM fails due to dense matrix operations and corresponding memory exhaustion.
• Iteration Behavior: Despite GTSAM’s lower iteration counts (due to exact subproblem solutions via direct solvers), total wall time is higher because of expensive direct solver steps. The proposed iterative matrix-free scheme achieves better time-to-solution.

These results are robust across random initializations and are aided by the benign optimization landscapes of the problem formulations (allowing certificate-based global optimality validation).

Implications and Future Directions

The key implication is that large-scale robotic-perception NLS problems with separable structure and gauge symmetry benefit disproportionately from this sparsity-preserving VarPro scheme. By integrating matrix-free Schur operators, it is possible to handle orders of magnitude larger instances using iterative solvers, reducing both runtime and memory requirements. This addresses primary bottlenecks in large-scale SLAM and SfM deployments.

On a theoretical level, the work provides clear guidance for practitioners on when and how variable elimination can be safely and efficiently applied, considering both the cost’s algebraic structure and the underlying graphical modeling.

For future research, this framework suggests several directions:

• Extension to nonlinear or inhomogeneous residuals, potentially by linearization at each iteration;
• Automated detection and exploitation of separable structures in generic factor graph solvers;
• Integration with distributed or parallelized optimization schemes for massive-scale multi-agent or decentralized systems.

Conclusion

This paper presents a principled, efficient solution strategy for separable NLS problems pervasive in robotic perception, overcoming longstanding challenges associated with gauge symmetries and computational bottlenecks of dense matrix operations. The proposed matrix-free variable projection operator consistently outperforms established solvers in both speed and memory footprint, offering a robust tool for future advances in scalable real-world perception systems. An open-source C++ implementation and extensive benchmarks further enhance its utility and reproducibility.