Model Contraction Optimization

• Model contraction optimization involves applying contraction theory from dynamical systems to guarantee exponential convergence and robust error bounds in various applications.
• It optimizes tensor network computations by designing contraction paths that minimize memory footprint and floating-point operations, thereby improving computational efficiency.
• In control and neural network compression, contraction-based methods simplify model predictive control constraints and enable alternating minimization for improved stability and performance.

Model contraction optimization refers to a class of techniques and theoretical frameworks that employ contraction principles—often from dynamical systems or metric analysis—to develop, analyze, and optimize models in settings ranging from control and optimization to tensor network contractions and neural network compression. In the context of systems, control, and optimization, contraction theory provides sufficient conditions for global incremental stability by ensuring that distances between trajectories contract over time, leading to robust convergence and strong error bounds. In high-dimensional computation (such as tensor networks), contraction optimization targets the ordering and structure of intermediate computations to minimize time and memory complexity. The contraction paradigm unifies and informs stability analysis, computational efficiency, robustness, and design of algorithms across multiple technical domains.

1. Fundamental Principles of Contraction-Based Optimization

Contraction theory studies the behavior of the distance between solutions of a dynamical system. Let $ẋ = f(x,t)$ be a (possibly time-varying) system. The system is said to be (strongly) contracting in a metric induced by $M(x)$ if, for all $x$ and $t$,

$$\frac{\partial f}{\partial x}^\top M + M \frac{\partial f}{\partial x} + \dot{M} \leq -2\lambda M$$

for some $\lambda > 0$, where $M$ is a positive-definite matrix function. This condition guarantees that for any two trajectories $x_1(t)$ and $x_2(t)$,

$$\|x_1(t) - x_2(t)\|_M \leq e^{-\lambda (t-s)} \|x_1(s) - x_2(s)\|_M$$

exponential convergence regardless of initial condition.

In optimization, contraction-based methods extend to discrete-time systems, variational optimality conditions, and PDEs. In model predictive control (MPC), contraction–based constraints or penalties replace classical terminal set and cost constructions to guarantee closed-loop stability (Alamir, 2016).

For high-dimensional computations such as tensor network contraction, model contraction optimization refers to searching for contraction trees or sequences minimizing peak intermediate tensor size (space), total floating-point operations (FLOPs, time), or approximation error, often relying on hyper-optimization over parameterized heuristics or randomized algorithms (Gray et al., 2020, Gray et al., 2022, Orgler et al., 2024).

2. Contraction Theory in Control and Model Predictive Optimization

Singularly-perturbed or multi-timescale systems are prime candidates for contraction-theoretic analysis. A general two-time-scale system reads:

$M(x)$0

where $M(x)$1 and $M(x)$2 are slow and fast variables, and $M(x)$3, $M(x)$4 represent exogenous inputs. The contraction framework provides explicit upper bounds on the difference between the full and reduced models in terms of contraction rates ($M(x)$5), Lipschitz constants, the parameter $M(x)$6, and disturbance magnitudes (Cothren et al., 2023):

$M(x)$7

When applied to online feedback optimization (OFO), where a fast plant is coupled with a (typically slower) gradient-flow controller, contractivity of both subsystems yields tight tracking error bounds to the time-varying optimizer, with explicit dependencies on controller gains and timescale separation. Contractivity of the closed-loop is linked to the negativity of a Metzler gain matrix $M(x)$8, whose spectral abscissa gives the closed-loop contraction rate and, thus, directly informs system design and gain selection (Cothren et al., 2023).

In MPC, contraction-based approaches enable the removal of explicit stabilizing terminal constraints. Instead, contraction is enforced via multi-step Lyapunov penalties in the cost, leading to short prediction horizons, improved feasibility in perturbed scenarios, and robust practical stability (Alamir, 2016, Polver et al., 4 Feb 2025, McCloy et al., 2022, Su et al., 2019).

Table: Contraction-Based MPC Formulations

Reference	Key Feature	Convergence Mechanism
(Alamir, 2016)	No terminal constraint	Multi-step contraction in stage cost
(Polver et al., 4 Feb 2025)	Robust under perturbation	Tube sets, contraction metric, flexible horizon
(McCloy et al., 2022)	Multi-timescale	Segment-wise contraction with nonuniform grid
(Su et al., 2019)	Distributed, inexact	Primal-dual gradient, contraction proof

3. Model Contraction Optimization in Tensor Networks

In computational disciplines, model contraction optimization most often refers to the search for optimal contraction paths (orderings) in tensor networks, with profound implications for both classical and quantum simulation.

A tensor contraction path is specified by a rooted binary tree or an ordered list of pairwise contractions. The total computation (cost $M(x)$9) and peak space (memory $x$0) are defined as:

$x$1

Optimization over contraction orderings is NP-hard, so practical algorithms combine greedy search, randomized tree sampling, hypergraph partitioning, and Bayesian hyper-optimization. Randomized and hyper-optimized protocols—for example, the Hyper-Par driver with Bayesian parameter tuning—have led to dramatic reductions in contraction complexity, enabling the simulation of systems such as random quantum circuits with speedups exceeding $x$2 over pre-existing heuristic methods (Gray et al., 2020, Gray et al., 2022, Orgler et al., 2024).

Multi-cost-function greedy methods further improve path discovery by selecting among competing cost heuristics at runtime, yielding orders-of-magnitude improvements in both solve time and FLOPs, especially for challenging instances in quantum simulation, model counting, and LLMs (Orgler et al., 2024).

Graph-theoretic variants recast contraction optimization as tree or carving decomposition minimization. For tensor networks with planar structure graphs, cubic-time optimal algorithms exist (Dudek et al., 2019). For general graphs, tree decompositions of the line graph enable near-optimal contraction tree construction, and hybrid factorization methods handle high-rank tensors.

Table: Tensor Network Contraction Optimization Approaches

Strategy	Cost metric	Optimization Algorithm	Empirical Advantage
Greedy heuristic	Peak size, flops	O($x$3)	Fast for small/medium networks
Randomized/HPO	Peak size/flops	Stochastic + Bayesian	Near-optimal paths at large scale
Graph decompositions	Max-rank	Tree/carving dec.	Theoretically optimal for planar/structured

4. Contraction-Based Optimization in Learning and Compression

Contraction optimization also arises in neural network compression via constrained optimization frameworks, where the search space is expressed as the sum of compressed components (e.g., quantized, low-rank, pruned layers) and an alternating minimization procedure is used to find the best additive decomposition subject to the model contraction constraints (Carreira-Perpiñán et al., 2021):

$x$4

where each $x$5 is a compression manifold. The alternating "LC" (Learning–Compression) algorithm alternates between unconstrained learning and block-wise best projection onto the sum-constrained manifolds, yielding models with strictly improved accuracy-entropy tradeoffs compared to single-technique compression.

The contraction principle extends to training dynamics as well: contraction-conditioned neural network filters enforce Lyapunov-like inequalities to avoid estimation or generative model collapse, even under constant sample size settings, by penalizing deviations from contraction in the loss function (Han et al., 30 Nov 2025).

5. Methodological Implications and Systematic Design

Contraction optimization enables:

• Explicit rate and robustness bounds: Contraction rates, Lipschitz constants, timescale parameters, and gains contribute to quantitative merging of control performance and implementation cost (Cothren et al., 2023).
• Decoupled error analysis: Bounds on slow and fast subcomponents are provided independently, aiding in modular system design and analysis.
• Tunable trade-offs: Timescale separation ($x$6), controller gain, and approximation precision can be systematically adjusted based on contraction matrix negativity or spectral abscissae (e.g., Metzler gain matrices) with transparent, computable thresholds (Cothren et al., 2023).
• Algorithmic simplification: Terminal constraints and endpoint Lyapunov arguments can often be omitted in favor of contraction metrics and penalties (Alamir, 2016, Polver et al., 4 Feb 2025), leading to shorter optimization horizons and easier implementation.
• Robustness and distributed computation: The contraction property is key for robust MPC under bounded disturbances (Polver et al., 4 Feb 2025) and guarantees exponential convergence in distributed primal-dual optimization, even under early (inexact) termination, as constraint tightening can be aligned with contraction margins (Su et al., 2019).
• Flexible approximation: In tensor contraction, hyper-optimization over contraction trees or compression budgets gives explicit cost–error trade-offs and generalizes to diverse network topologies and connectivities (Gray et al., 2020, Gray et al., 2022).

6. Connections to Broader Optimization Theory

In continuous optimization, contraction methods generalize classic global optimization procedures by constructing nested closed sets that shrink toward the minimizer set, subject to explicit model error criteria

$x$7

Under appropriate hierarchical low-frequency dominance (HLFDF) and regularity assumptions, such contraction methods achieve linear or polynomial convergence over successive contractions, distinguishing problems solvable in logarithmic or polynomial time from intractable cases. Contractibility serves as a complement to classical smoothness-based assumptions (Luo et al., 2019).

In the discretization and analysis of optimization flows (for instance, deriving Nesterov's accelerated method), contraction theory provides a unified justification for the stability and convergence rate of both continuous-time flows and their Euler discretizations, directly linking Lyapunov exponents and contraction rates to optimization algorithm performance (Cisneros-Velarde et al., 2021).

---

References

• (Cothren et al., 2023) Online Feedback Optimization and Singular Perturbation via Contraction Theory
• (Alamir, 2016) A New Contraction-Based NMPC Formulation Without Stability-Related terminal Constraints
• (Gray et al., 2020) Hyper-optimized tensor network contraction
• (Gray et al., 2022) Hyper-optimized approximate contraction of tensor networks with arbitrary geometry
• (Polver et al., 4 Feb 2025) Robust contraction-based model predictive control for nonlinear systems
• (McCloy et al., 2022) A Contraction-constrained Model Predictive Control for Multi-timescale Nonlinear Processes
• (Su et al., 2019) Distributed Model Predictive Control Under Inexact Primal-Dual Gradient Optimization Based on Contraction Analysis
• (Han et al., 30 Nov 2025) Preventing Model Collapse via Contraction-Conditioned Neural Filters
• (Cisneros-Velarde et al., 2021) A Contraction Theory Approach to Optimization Algorithms from Acceleration Flows
• (Luo et al., 2019) Contraction methods for continuous optimization
• (Carreira-Perpiñán et al., 2021) Model compression as constrained optimization, with application to neural nets. Part V: combining compressions
• (Dudek et al., 2019) Efficient Contraction of Large Tensor Networks for Weighted Model Counting through Graph Decompositions
• (Orgler et al., 2024) Optimizing Tensor Contraction Paths: A Greedy Algorithm Approach With Improved Cost Functions