A Differentiable Interior-Point Method in Single Precision

Abstract: Primal-dual interior-point methods solve constrained convex optimization problems to tight tolerances with speed and robustness. Their solutions are also efficiently differentiable with respect to the problem data through the implicit function theorem. However, the standard treatment of primal-dual complementarity makes the underlying linear systems increasingly ill-conditioned near the solution. While this ill-conditioning is often benign in double precision, it can be catastrophic in single precision, preventing interior-point methods from fully exploiting the accelerated hardware that underpins modern machine learning. This paper introduces a differentiable interior-point method designed for low-precision arithmetic. By using an alternative complementarity representation, we ensure that the underlying linear systems remain spectrally bounded -- even near the solution -- a property that is essential for computing accurate gradients and avoiding arithmetic exceptions. As a result, our method enables interior-point solvers to reliably solve and differentiate optimization problems in single precision that were previously confined to double precision. We demonstrate the approach through an ablation study against the standard interior-point formulation and applications in bilevel and end-to-end learning settings where differentiating through constrained optimization is essential. The source code is available at https://github.com/qpax-solver/qpax.

• The paper introduces a novel implicit complementarity approach that stabilizes KKT systems in single precision for quadratic programming.
• It employs the implicit function theorem to compute efficient Jacobian-vector products, ensuring high-gradient fidelity under tight relaxations.
• Experimental results demonstrate robust performance and improved memory scaling in applications like bilevel optimization and multi-agent safety filters.

Differentiable Interior-Point Methods in Single Precision: Robust Optimization and Gradient Computation

---

Motivation and Context

Constrained convex optimization is foundational across control, simulation, operations research, and machine learning. In recent years, differentiable convex optimization pipelines have become critically important as machine learning frameworks increasingly embed optimization layers within neural architectures (2605.17913). Primal-dual interior-point methods (IPMs) are the gold standard for solving these convex programs due to their speed, robustness, and rigorous theoretical guarantees.

However, the classical treatement of complementarity in IPMs makes the underlying KKT linear systems ill-conditioned near the solution. While benign in double-precision (f64), this is catastrophic in single-precision (f32), which is the preferred arithmetic for modern ML hardware due to its throughput and memory efficiency. This mismatch restricts the integration of differentiable interior-point solvers within high-performance ML workflows.

This paper proposes a new differentiable IPM for quadratic programming (QP) that exploits an implicit complementarity representation to yield spectrally bounded and structurally robust KKT systems, enabling reliable solution and differentiation in single precision.

---

Quadratic Programming and the Standard Interior-Point Paradigm

The paper begins with a rigorous definition of the canonical quadratic program (QP), including slack reformulation and KKT optimality conditions. In the standard primal-dual approach, complementarity is enforced explicitly and perturbed using a central path parameter $\kappa$, leading to a KKT system whose conditioning collapses as the solution converges—particularly as $s \odot z \to 0$ for active constraints. While this is manageable in double precision, it severely limits accuracy and robustness in single precision.

Differentiability is addressed using the implicit function theorem, which allows efficient computation of derivatives at the solution without unrolling solver iterations. The paper employs relaxed optimization (evaluating gradients at $\kappa_\mathrm{relax}$) to recover informative gradients even in nonsmooth regimes.

---

Spectrally Bounded Implicit Complementarity Approach

To resolve structural ill-conditioning, the authors adopt an implicit complementarity scheme. An auxiliary variable $v$ and a retraction map $b_\kappa(v)$ are introduced to enforce $z \odot s = \kappa$ by construction. The softplus retraction, $b_\kappa(v) = \frac{v + \sqrt{v^2 + 4\kappa}}{2}$, yields diagonal blocks in the KKT system bounded strictly between $0$ and $1$.

The resulting condensed KKT system (with $Q-G^\top G$ and $s \odot z \to 0$0 blocks) is symmetric and spectrally bounded, eliminating the eigenvalue divergence that destabilizes the standard explicit approach and mitigating arithmetic exceptions in f32:

Figure 1: Implicit complementarity enables accurate and robust single-precision gradients. Left: As problem dimension and active-set size increase, the implicit method maintains low gradient error and remains fully robust, whereas the explicit formulation fails $s \odot z \to 0$1 of the time. Right: As $s \odot z \to 0$2 decreases, the implicit method continues to track the relaxation-limited gradient accuracy, whereas the explicit f32 formulation plateaus in accuracy and exhibits escalating failure rate.

---

Differentiation Pipeline and Algorithmic Framework

The method comprises three stages: (i) solve QP to tight tolerance using implicit complementarity, (ii) relax the solution by tracing the central path to prescribed $s \odot z \to 0$3 for informativeness, and (iii) compute Jacobian-vector products using the implicit function theorem at the relaxed point. The condensed KKT factors are cached and reused, yielding highly efficient forward and backward passes in automatic differentiation frameworks.

Algorithmic details are provided, including explicit pseudocode for QP solving, relaxation, and backward differentiation.

---

Experimental Evaluation and Numerical Results

The approach is benchmarked on synthetic and real-world tasks:

Gradient Fidelity and Robustness

Implicit formulation achieves high-gradient fidelity and $s \odot z \to 0$4 failure rate in f32, even as active sets and dimensionality grow, while explicit f32 exhibits sharp error increase and frequent NaN failures—especially under tight relaxation. These results are robust across random seeds and parameter sweeps.

Bilevel Trajectory Optimization

Application to bilevel optimization for robotics (trajectories through forest/office environments) demonstrates that implicit f32 matches f64 in performance and reliability, whereas explicit f32 gradients are unusable or fail entirely for tight relaxation. The consistency extends across environments and relaxation settings.

Figure 2: Bilevel trajectory optimization. Implicit f32 remains stable at tight relaxations and matches f64 performance, whereas explicit f32 becomes unreliable.

End-to-End Learning of Multi-Agent Safety Filters

In high-dimensional multi-agent safety filter learning, implicit f32 matches f64 stability and training efficiency, while explicit f32 fails for more than one agent. Implicit f32 enables double the maximum batch size, leveraging GPU memory to its full potential.

Figure 3: End-to-end learning of a multi-agent safety filter. Implicit f32 matches f64 training stability (unlike explicit f32, which fails entirely) and allows for double the maximum batch size.

---

Implications, Limitations, and Future Directions

This work unlocks the full potential of differentiable IPMs in single-precision arithmetic, aligning the optimization workflow with the requirements of modern ML accelerators. Practically, it enables large-batch end-to-end learning architectures, differentiable trajectory planning, and real-time safety filtering in scalable, high-constraint environments.

A key limitation is the focus on QPs; extension to other conic programs and nonlinear interior-point methods remains an open challenge. Integration with mixed-precision solvers, further hardware exploration, and investigation of adaptive retraction maps could broaden the applicability. The ability to reliably differentiate through optimization layers in f32 paves the way for deeper coupling of optimization and neural modeling in future AI systems.

---

Conclusion

The implicit complementarity formulation provides a spectrally bounded, numerically robust foundation for differentiable quadratic programming in single precision. Empirical results highlight significant improvements in gradient fidelity, solver reliability, and memory scaling over the explicit approach. This paradigm shift is essential for embedding constrained optimization into the machine learning stack at scale, offering both practical and theoretical advances for optimization-aware AI.