---
title: 'SLSQP: Sequential Least Squares Programming'
url: https://www.emergentmind.com/topics/sequential-least-squares-quadratic-programming-slsqp
type: topic
---

# SLSQP: Sequential Least Squares Programming

Sequential Least Squares Quadratic Programming (SLSQP) is a widely adopted algorithm for solving nonlinear programming (NLP) problems subject to both equality and inequality constraints. It iteratively solves a sequence of quadratic or least-squares subproblems, employing quasi-Newton updates, constraint management, and line search techniques to ensure global convergence. Modernizations such as PySLSQP and algorithmic improvements like I-SLSQP have further enhanced its transparency, robustness, and applicability to ill-conditioned and medium-scale optimization problems [2408.13420][2402.10396].

## 1. Mathematical Structure and Algorithmic Core

At major iteration $k$, SLSQP generates a QP subproblem using the current iterate $x_k$, Lagrangian gradient $g_k$, and a symmetric positive-definite Hessian approximation $B_k$:
\[
\begin{array}{rl}
\min_{p\in\mathbb R^n} & \frac{1}{2}p^T B_k p + g_k^T p \\
\text{subject to} & A_{eq,k}p + c_{eq}(x_k) = 0 \\
& A_{ineq,k}p + c_{ineq}(x_k) \ge 0
\end{array}
\]
where $A_{eq,k}$ and $A_{ineq,k}$ are the Jacobians of the equality and inequality constraints, respectively.

The corresponding Karush-Kuhn-Tucker (KKT) optimality system enforces stationarity, primal feasibility, dual feasibility, and complementary slackness. If second derivatives are unavailable, SLSQP maintains $B_k\succ0$ through a limited-memory BFGS update:
\[
B_{k+1} = B_k + \frac{y_k y_k^T}{y_k^T s_k} - \frac{B_k s_k s_k^T B_k}{s_k^T B_k s_k}
\]
for $s_k = x_{k+1}-x_k$ and $y_k = \nabla_x L(x_{k+1},\lambda_{k+1},\mu_{k+1}) - \nabla_x L(x_k,\lambda_k,\mu_k)$, with suitable damping if $y_k^T s_k\le0$ [2408.13420].

## 2. Subproblem Solution and Enhanced Variants

Standard SLSQP forms the QP or, equivalently, a linear constrained least-squares (LSQ) system:
\[
\min_{d}\|R_k d + q_k\|_2^2 \qquad
\text{subject to}\quad V_h(x_k)^T d + h(x_k) = 0,\; V_g(x_k)^T d + g(x_k)\ge 0
\]
with $B_k = R_k^T R_k$. When the LSQ subproblem is inconsistent, improved variants like I-SLSQP apply hybrid relaxations:
- **RLSQ1**: Modified Powell relaxation introducing a scalar slack $\delta$.
- **RLSQ2**: Nowak-type relaxation using slack vectors with strong penalties.

I-SLSQP dynamically switches between these relaxations depending on feasibility and conditioning, returning the first viable step direction [2402.10396].

In traditional implementations, dual LSQ solvers leverage nonnegative least squares (NNLS). However, in the presence of tiny denominators (specifically, small $r_{n'+1}$ in $\frac{-r_i}{r_{n'+1}}$), numerical cancellation causes catastrophic search direction failures. When such pathologies are detected (e.g., ascending directions or abnormally large step norms), I-SLSQP falls back to a protected QP solver for the LSQ subproblem [2402.10396].

## 3. Globalization: Scaling, Derivative Estimation, and Merit Functions

Numerical stability is enhanced in PySLSQP by diagonal scaling of variables and constraints:
\[
x = D_x \tilde{x}; \;\; f(x) = D_f \tilde{f}(\tilde{x}); \;\; c(x) = D_c \tilde{c}(\tilde{x})
\]
with user-supplied scalers $D_x$, $D_f$, $D_c$. Gradients and steps are automatically scaled and unscaled during processing, improving performance on poorly conditioned problems [2408.13420].

When analytic derivatives are unavailable, forward finite-differences approximate gradients. PySLSQP supplies `finite_diff_abs_step` and `finite_diff_rel_step` options, monitoring function variation to adapt the step $h$ and mitigate cancellation, adapting as necessary down to machine precision.

For globalization, SLSQP employs a backtracking line search on an augmented $L_1$-merit function:
\[
\phi(x;\sigma) = f(x) + \sigma \sum_{i=1}^{m_{eq}} |c_{eq,i}(x)| + \sigma \sum_{i=1}^{m-m_{eq}} [\min(0, c_{ineq,i}(x))]
\]
with adaptive $\sigma$, enforcing sufficient decrease [2408.13420].

## 4. Implementation: PySLSQP Modernization and API

PySLSQP wraps the original SLSQP Fortran kernel from Kraft, using Meson and f2py to produce a compiled `_slsqp.so` extension. The Python front end (`pyslsqp.optimize`) manages argument parsing, scaling, finite-difference evaluation, calls into the Fortran driver, and handles data saving and visualization.

The API accepts wide-ranging user options, including warm/hot restarts, variable and constraint scaling, finite-difference tunings, and selection of which algorithmic internals to save:

```python
from pyslsqp import optimize
res = optimize(
    x0, obj=objective, con=constraints, jac=jacobian,
    meq=1, xl=x_lower, xu=x_upper,
    finite_diff_abs_step=1e-6,
    x_scaler=10.0, obj_scaler=2.0, con_scaler=[1.0, 0.5],
    save_itr='major', save_vars=['majiter', 'x', 'objective'],
    save_filename="save_file.hdf5",
    visualize=True,
    visualize_vars=['objective', 'x[0]']
)
```

Critical features of PySLSQP include:
- Access and live storage of internal optimizer state (iterates, multipliers, BFGS diagonals, etc.).
- Seamless warm and hot restarts for rapid re-optimization.
- Live plotting and post-processing via Matplotlib using HDF5 files, with routines for loading and visualizing optimization histories.
- Diagnostic log files and rich tools for integrating into research workflows [2408.13420].

## 5. Robustness, Convergence, and Comparative Performance

I-SLSQP and PySLSQP introduce robust handling of ill-conditioned subproblems via their hybrid LSQ/QP solution and dual-LSQ failure safeguards. I-SLSQP in particular only resets the Hessian update when an ascent direction is encountered, avoiding premature loss of curvature information. Its two-group convergence testing (feasibility and optimality) ensures algorithmic termination iff stationarity or step tolerance is met.

In extensive computational experiments across 42 large-scale nonlinear process engineering instances, I-SLSQP was the only tested method to succeed in all cases, reliably overcoming infeasibility declarations that halted fmincon (MATLAB SQP) and IPOPT. PySLSQP—corresponding to the original Kraft/Schittkowski SLSQP—solved nearly all instances but prematurely terminated on two. I-SQP (improved QP-based SQP) performed efficiently on well-conditioned classes but was less robust on ill-conditioned problems.

Benchmarks for medium-scale problems (≈200 variables/constraints) demonstrate that PySLSQP converges within 200 function evaluations on optimal-control tasks where SNOPT, IPOPT, and SciPy’s Trust-Constr failed or returned infeasible solutions under identical evaluation budgets [2408.13420][2402.10396].

## 6. Practical Considerations, Limitations, and Outlook

PySLSQP provides a transparent, research-oriented interface for SLSQP, exposing all internal quantities, enabling reproducibility, and allowing customization through warm/hot restarts and tuning options. Its flexible scaling and robust finite-difference handlers address longstanding challenges of numerical instability and lack of transparency in black-box optimizers.

For well-conditioned or small to medium NLPs, I-SQP may deliver superior walltime performance due to lower overhead. For ill-conditioned and constraint-dominated problems, I-SLSQP and PySLSQP provide more reliable convergence and better minima, with only modest additional computational cost.

A plausible implication is that SLSQP, coupled with modern transparency and robustification strategies, remains state-of-the-art for medium-scale NLPs requiring constraint feasibility, flexible post-processing, and on-the-fly monitoring. The separation of algorithmic core (Fortran or compiled extension) and flexible Python front end provides an extensible base for future research in large-scale nonlinear optimization [2408.13420][2402.10396].

---

**References**  
- "PySLSQP: A transparent Python package for the SLSQP optimization algorithm modernized with utilities for visualization and post-processing" [2408.13420]  
- "Improved SQP and SLSQP Algorithms for Feasible Path-based Process Optimisation" [2402.10396]

Source: https://www.emergentmind.com/topics/sequential-least-squares-quadratic-programming-slsqp