---
title: 'NSPO: Null-Space Constrained Policy Optimization'
url: https://www.emergentmind.com/topics/null-space-constrained-policy-optimization-nspo
type: topic
---

# NSPO: Null-Space Constrained Policy Optimization

Null-Space Constrained Policy Optimization (NSPO) is a family of geometric algorithms for policy optimization under explicit linear constraints. The core principle is to confine gradient or Newton-style policy updates to the null space of specified constraint matrices, ensuring that policy parameters are only allowed to vary in directions which do not compromise predefined invariants or capabilities. This framework is applicable both in feedback control synthesis under linear constraints and, more recently, in reinforcement learning settings for large-language-model (LLM) alignment, where it enables safety alignment without sacrificing general-purpose abilities.

## 1. Mathematical Foundation and Constraints

NSPO requires formalization of the constrained policy space as an intersection of a feasible set (e.g., stabilizing policies or general-capability-preserving policies) and an affine subspace defined by explicit linear constraints. In the context of Schur-stable feedback synthesis, this is represented as follows: let $K \in \mathbb{R}^{m \times n}$ be a feedback gain, $S = \{K : \rho(A + BK) < 1\}$ the manifold of Schur-stable controllers for system matrices $(A, B)$, and a family of linear constraints $C\,\mathrm{vec}(K) = d$ for $C \in \mathbb{R}^{p \times (mn)}$, $d \in \mathbb{R}^p$. The feasible manifold is
\[
M = S \cap \{K : C\,\mathrm{vec}(K) = d\}
\]
where $M$ is an embedded submanifold of dimension $mn-p$ under standard rank conditions [2201.11157].

In safety alignment for LLMs, the constraint is that parameter updates $\theta$ should not interfere with general-capability representations as encoded in hidden-state vectors $K \in \mathbb{R}^{d \times N}$. The linear constraint is that $\theta$ is only updated in directions orthogonal to the column span of $K$ [2512.11391].

## 2. Null-Space Projections and Geometry

The null-space projector is constructed to ensure that updates remain feasible. Algebraically, for a set of linear constraints $C\,\mathrm{vec}(K) = d$, the tangent directions are the kernel of $C$, with the projection $\Pi = I - C^\top (CC^\top)^{-1} C$. For LLM alignment, the null space is defined by the singular value decomposition (SVD) of the matrix $K K^\top = U \Lambda U^\top$, with the null space spanned by columns of $U_0$ where eigenvalues $\lambda_i$ are below a threshold, yielding a projector $P = U_0 U_0^\top$ [2512.11391].

These projections guarantee that any update $G$ to the policy satisfies $C\,\mathrm{vec}(K+G) = d$ or, in the LLM case, that the hidden-state feature representations for the general capability data remain invariant under the new policy.

## 3. NSPO Optimization Algorithms

### Feedback Controller Synthesis

For linearly constrained quadratic-regulator design, a Riemannian metric is adopted on $S$, induced by the Lyapunov solution $Y_K$. The projected (constrained) Riemannian gradient and Hessian are:
- $\mathrm{grad}\,h|_K = \Pi\,\mathrm{grad}\,f|_K$
- $\mathrm{Hess}\,h|_K[U] = \Pi\,\mathrm{Hess}\,f|_K[U]$
  
The update step solves (in vector notation):
\[
\Pi[\mathrm{Hess}\,f|_{K_t}]\Pi\,\mathrm{vec}(G_t) = -\Pi\,\mathrm{vec}(\mathrm{grad}\,f|_{K_t})
\]
with $K_{t+1} = K_t + \eta_t G_t$, and the step size $\eta_t$ is capped by a stability certificate to ensure Schur stability [2201.11157].

### LLM Safety Alignment

For RL-based safety alignment, the standard policy gradient $g(\theta)$ is projected:
\[
g_{\mathrm{NSPO}}(\theta) = P\,g(\theta)
\]
where $P$ projects onto the null space of general-capability representations $K$. Iteratively, $\theta$ is updated as $\theta \leftarrow \theta - \eta\,g_{\mathrm{NSPO}}$ [2512.11391].

Key theoretical results include:
- **Gradient Norm Bound**: $\|g_{\mathrm{NSPO}}\|_2 \leq \|g\|_2$ (since $P$ is an orthogonal projector).
- **Descent Guarantee**: For sufficiently small step size $\eta > 0$, $J(\theta - \eta g_{\mathrm{NSPO}}) \leq J(\theta)$ [2512.11391].

## 4. Algorithmic Procedures and Pseudocode

The canonical QRNPO/NSPO algorithm for feedback synthesis is as follows [2201.11157]:

1. Input system $(A, B)$, constraints $C\,\mathrm{vec}(K) = d$, cost function $f$, and initial $K_0$.
2. For $t = 0, 1, \dots$ until convergence:
    - Compute ambient gradient and Hessian.
    - Project both gradient and Hessian with $\Pi$.
    - Solve for the projected Newton direction.
    - Compute stability-certified step size $s$.
    - Update $K_{t+1} = K_t + \eta_t G_t$, with $\eta_t = s$.

For LLM safety alignment [2512.11391]:

1. Extract hidden states $K$ from base model on general-capability data.
2. Compute $K K^\top$ and its SVD to assemble $P$.
3. Initialize $\theta = \theta_0$.
4. For each RLHF iteration:
    - Sample safety minibatch and compute policy gradient $g$.
    - Project: $g_{\mathrm{NSPO}} = P g$.
    - Update: $\theta \leftarrow \theta - \eta g_{\mathrm{NSPO}}$.

## 5. Empirical Findings and Theoretical Guarantees

In the LLM alignment context, NSPO achieves:
- State-of-the-art safety (e.g., lowest Attack Success Rates across seven safety benchmarks), with 5–15 point reductions relative to previous methods.
- General capabilities (math, code, instruction following) are preserved, with $<1\%$ absolute drop, matching baselines requiring mixed-task data.
- Data efficiency: only $40\%$ of publicly available safety data required; no need to interleave general-task examples.
- Computational overhead is nominal: SVD of size $d \times d$ offline; per-step projection $O(d^3 + n^2 d + n d^2)$ given $d \ll n$; extra $O(d^2)$ GPU memory, typically offloaded [2512.11391].

For linear-quadratic regulators, NSPO attains local quadratic convergence rates under standard second-order conditions, outpacing projected-gradient methods which yield only $O(1/\epsilon)$ rates [2201.11157].

| Context     | Constraint Type            | Projection Matrix | Core Theoretical Guarantee         |
|-------------|---------------------------|-------------------|------------------------------------|
| LQR         | Linear in $\mathrm{vec}(K)$ | $\Pi$             | Local quadratic convergence        |
| LLM Alignment | Column span of $K$         | $P$               | Descent, invariance of K          |

## 6. Applications and Extensions

NSPO provides a principled solution for a wide spectrum of constrained control and learning problems where strict invariance or preservation properties are required:
- **Feedback Control**: Contemporary advances in constrained synthesis over the Schur-stabilizing manifold permit efficient and geometrically well-founded solutions without recourse to manifold retraction or exponential mapping [2201.11157].
- **LLM Safety Alignment**: NSPO enables reinforcement learning from human feedback to improve safety without degradation of general-purpose skills, directly addressing the so-called alignment tax problem by projecting updates into capability-preserving subspaces [2512.11391].

A plausible implication is that NSPO can be extended to any learning setting where certain invariances or constraints must be strictly enforced, including robust control, constrained reinforcement learning beyond LLMs, and safe learning under domain-specific operational constraints.

## 7. Comparison with Projected and Unconstrained Methods

NSPO delivers strict constraint satisfaction by construction. Standard projected-gradient methods update parameters indiscriminately and then project back onto the constraint set, generally resulting in slower, sublinear convergence ($O(1/\epsilon)$) and potential infeasibility between steps. NSPO leverages the geometry of the constraint manifold, with second-order updates achieving local quadratic rates and ensuring iterates remain feasible after each step. In language model alignment, prior RLHF approaches require continual mixing of general-task data; NSPO decouples general capability preservation from this requirement through explicit projection, confining safety-driven updates to the null space of those capabilities [2201.11157, 2512.11391].

Source: https://www.emergentmind.com/topics/null-space-constrained-policy-optimization-nspo