---
title: Iterative Null-space Projection (INLP)
url: https://www.emergentmind.com/topics/iterative-null-space-projection-inlp-cff768dc-8174-44f5-a436-d63426952b3d
type: topic
---

# Iterative Null-space Projection (INLP)

Searching arXiv for recent and foundational papers on INLP and closely related work.
Iterative Null-space Projection (INLP) is a post-hoc, model-agnostic method for removing linear information about a designated target attribute from learned representations while preserving as much task-relevant information as possible. Introduced for guarding protected attributes in neural representations, INLP repeatedly trains a linear classifier to predict the attribute of interest and then projects the representations onto the null-space of the classifier’s row space, thereby eliminating the linear directions used by that classifier [2004.07667]. In the NLP literature, the method has been used for bias mitigation, fairness interventions, and amnesic probing; later work has also scrutinized its geometric side effects and proposed single-projection or closed-form alternatives [2212.04273; 2506.11673].

## 1. Definition and mathematical construction

INLP assumes a representation vector $h \in \mathbb{R}^d$ or a matrix of representations $X \in \mathbb{R}^{n \times d}$, together with a protected or target attribute $z$ with $k$ possible classes. Its objective is to transform the representation so that, after transformation, no linear classifier should be able to predict $z$ better than chance from the representation [2004.07667].

At each iteration, a strong linear classifier is trained on the current representations to predict the target attribute. Common choices are multiclass logistic regression with $L2$ regularization and linear SVM. For softmax classifiers,
$$
p(z=i \mid h) = \mathrm{softmax}_i(W h + b).
$$
If the classifier has weight matrix $W \in \mathbb{R}^{k \times d}$, INLP removes the row space of $W$ by projecting onto its orthogonal complement. Using an orthonormal basis $Q$ for the row space,
$$
P = I_d - Q Q^T.
$$
Using $W$ directly, assuming full row rank,
$$
P = I_d - W^T (W W^T)^{-1} W,
$$
and when $W W^T$ is rank-deficient,
$$
P = I_d - W^T (W W^T)^{\dagger} W.
$$
With thin SVD $W = U \Sigma V^T$, letting $V_r$ denote the right singular vectors corresponding to nonzero singular values,
$$
P = I_d - V_r V_r^T.
$$
The transformed representation is then $h' = P h$, or in matrix form,
$$
X_{t+1} = X_t P_t.
$$
The total projection after $K$ iterations is the composition
$$
P_{\text{total}} = P_K P_{K-1} \cdots P_1.
$$
Efficient application avoids forming dense projectors and instead uses
$$
X_{t+1} = X_t - (X_t V_r) V_r^T,
$$
with cost $O(n d r)$ per iteration [2004.07667].

A central guarantee is immediate for the trained classifier: projecting onto the exact null-space of the row space ensures that, for any $h$,
$$
W(P h)=0
$$
up to numerical precision, so the specific classifier’s logits reduce to the bias term and become independent of $h$ [2004.07667].

## 2. Iterative procedure, stopping, and implementation

The basic INLP loop initializes $X_1 \leftarrow X$, then alternates between fitting a linear classifier on $(X_t, z)$, computing the projection matrix from the classifier’s row space, projecting the current representations, and evaluating whether the target attribute remains linearly recoverable [2004.07667]. Retraining after each projection is essential because a single projection may fail to remove residual linear signals not captured by the first classifier, for example because of regularization, class imbalance, or a weak probe [2004.07667].

Stopping is typically based on one of three criteria: probe performance reaching near chance, a fixed number of iterations, or a rank or utility budget. Chance-level stopping is often phrased as validation accuracy approaching $1/k$, or balanced accuracy approaching $1/k$ in multiclass settings [2004.07667]. Validation-based early stopping is also used when downstream utility begins to degrade while fairness gains plateau [2004.07667].

Several implementation details recur across the literature. SVD is preferred for constructing the row-space basis and for numerical stability; pseudoinverses are used when $W W^T$ is rank-deficient; and accumulated bases may be re-orthonormalized with procedures such as Gram–Schmidt or QR to prevent drift [2004.07667]. For large $n$, batching and low-rank application of the projector are standard efficiency measures [2004.07667].

In applied work on translationese debiasing, the procedure is instantiated almost literally. Linear classifiers are trained on fixed sentence or word embeddings, projections are computed with the SVD-based INLP implementation, and iteration continues until the translationese classifier reaches chance accuracy, approximately $50\%$ on balanced data [2205.08001]. In that study, reported iteration counts for NLI debiasing were $nclfs=45$ for sentence-level SNLI, $nclfs=34$ for aligned word-level debiasing, and $nclfs=35$ for joint word-level debiasing [2205.08001].

## 3. Guarantees and limitations of linear erasure

INLP removes linear information. More precisely, it ensures that the transformed representation is not linearly separable along directions spanned by the accumulated row spaces of the trained probes [2004.07667]. This is the sense in which the method “removes” a protected attribute: it eliminates linear signals of that attribute from the representation as seen by the sequence of linear classifiers used during the procedure [2004.07667].

The method does not guarantee removal of nonlinear information. Nonlinear classifiers, quadratic interactions, or other nonlinear functionals of the representation may still recover the target attribute after INLP [2004.07667]. This limitation is explicit in both the original formulation and later critiques. Work on amnesic probing argues that, although INLP is commonly used to remove a target property, its repeated projections can introduce collateral modifications to the overall space, complicating causal interpretation of downstream performance changes [2506.11673].

Dimensionality reduction is intrinsic to the method. If iteration $t$ removes a subspace of rank $r_t = \mathrm{rank}(W_t)$, then the effective dimension is reduced by the rank of the union of the removed subspaces, and in the multiclass case $r_t \le \min(k,d)$ [2004.07667]. This explains why aggressive use of INLP can alter geometry more broadly than the target attribute alone.

A worked binary example makes the construction explicit. For a classifier weight vector
$$
w = [1,1,0]^T,
$$
the projection matrix is
$$
P = I_3 - \frac{w w^T}{w^T w}
= \begin{bmatrix}
1/2 & -1/2 & 0 \\
-1/2 & 1/2 & 0 \\
0 & 0 & 1
\end{bmatrix}.
$$
For
$$
h=[2,0,3]^T,
$$
the projected vector is
$$
h' = P h = [1,-1,3]^T,
$$
and $w^T h' = 0$, so that classifier’s logit becomes independent of $h$ [2004.07667].

## 4. Empirical uses in bias mitigation and representation editing

The original evaluation of INLP focused on bias and fairness use-cases. In word embeddings, INLP reduces the ability of linear gender classifiers to predict gender-associated signals and alters nearest-neighbor structure so that words whose neighborhoods are dominated by gendered terms become more neutral after projection [2004.07667]. Appendix examples in that work identify strong shifts for gendered terms such as “mr., mrs., his, her, he, she,” as well as profession-associated words [2004.07667].

In text classification, the method has been applied to bag-of-words or neural sentence representations for multi-class profession classification with gender as the protected attribute. The reported pattern is a strong drop in attribute-prediction accuracy from high pre-INLP accuracy to near chance after $K$ iterations, alongside improved demographic parity and more balanced macro-F1 across protected groups, with modest decreases in downstream accuracy or F1 [2004.07667].

A distinct application concerns translationese. In "Towards Debiasing Translation Artifacts" [2205.08001], INLP is used to remove the signal distinguishing original from translated text from sentence and word embeddings. Sentence-level translationese classification accuracies before debiasing were reported as $0.64$–$0.72$ for fastText, $0.73$–$0.79$ for mBERT\_CLS, $0.82$–$0.86$ for mBERT\_pool, and $0.71$–$0.79$ for XLM-R CLS; after INLP debiasing, accuracy became $0.50$ in all cases [2205.08001]. At the word level, direct joint-space classification dropped from $0.91$–$0.98$ to $0.50$, and stepwise aligned-space classification dropped from $0.99$–$1.00$ to $0.50$ after INLP [2205.08001].

The same study evaluated downstream NLI using back-translated data. In the symmetric setting, sentence-level debiasing improved SNLI accuracy from $58.4 \pm 0.0$ on back-translated embeddings to $60.1 \pm 0.0$ after debiasing; in the asymmetric setting, it improved from $63.5 \pm 0.0$ to $64.9 \pm 0.0$ [2205.08001]. Word-level gains were smaller but positive in both joint and aligned setups [2205.08001]. The authors interpret these improvements as evidence that reducing translationese bias can mitigate train–test asymmetry induced by translation artifacts [2205.08001].

More recent work extends the projection idea beyond strictly linear ambient spaces. A 2026 recommendation paper proposes a kernelized INLP combined with a gated Mixture-of-Experts adapter for LLM-based recommender systems, where a closed-form projector is computed in an RFF-lifted space and then applied as a frozen buffer to backbone representations [2603.23780]. This suggests that the null-space projection paradigm has been adapted to settings where nonlinear attribute leakage is substantial, although the resulting procedure is no longer identical to the 2020 linear formulation.

## 5. Critiques, side effects, and proposed alternatives

A major line of subsequent work argues that multiple INLP iterations can damage non-target information. "Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection" [2212.04273] states that multiple projections increase the risk that information other than the target is negatively affected. That paper introduces Mean Projection (MP) and Tukey Median Projection (TMP) as single-projection alternatives and reports that one MP projection can achieve linear guarding with less impact on the overall space than many INLP iterations [2212.04273].

On GloVe gender-debiasing experiments, MP reduced linear classifier accuracy to $34.18\%$ in a single projection for the 3-class setting, whereas INLP required $12$ iterations to reach $34.9\%$; in the binary setting, MP reached $50.6\%$ in one step and INLP needed $14$ iterations to obtain $50.51\%$ [2212.04273]. Neighborhood stability also differed sharply: for $40$ random words, MP changed $8$ neighbors versus $43$ for INLP, and for $22$ gendered names, MP changed $24$ neighbors versus $56$ for INLP [2212.04273]. That paper further argues that many of the late-stage improvements attributed to INLP can be reproduced by random projections after an initial targeted step, implying that such gains may derive from generic dimensionality reduction rather than additional targeted attribute removal [2212.04273].

A more explicit critique appears in "Improving Causal Interventions in Amnesic Probing with Mean Projection or LEACE" [2506.11673]. There, INLP is described as a widely used removal technique in amnesic probing, but the paper reports that repeated classifier-dependent projections introduce random modifications to representations and reduce the method’s “targetedness” [2506.11673]. For non-masked BERT, INLP removed hundreds of directions and produced large rank reductions: directions removed were $779$ for dependency labels, $765$ for fine POS, and $240$ for coarse POS, with rank changes of $629$, $617$, and $198$, respectively [2506.11673]. Mean Projection and LEACE removed far fewer directions and preserved rank much better in the same setting [2506.11673].

The same paper reports low cosine similarity between original and modified representations under INLP—$0.31$ for dependency labels, $0.37$ for fine POS, and $0.80$ for coarse POS in non-masked BERT—whereas MP yielded $0.83$, $0.80$, and $0.91$, and LEACE yielded $0.92$, $0.95$, and $0.89$ [2506.11673]. In language modeling, INLP often failed the information control because matched random projections of the same cardinality sometimes harmed performance even more than amnesic INLP, especially for dependency and fine POS removal [2506.11673]. The paper therefore recommends MP by default and LEACE when stronger linear-erasure guarantees are required [2506.11673].

These critiques do not negate the original guarantee that the trained linear probe’s row space is removed. Rather, they delimit what that guarantee means operationally. The main dispute concerns whether repeated projection remains a clean intervention when the scientific goal is causal interpretation or fine-grained preservation of non-target information [2212.04273; 2506.11673].

## 6. Positioning, variants, and nomenclature

INLP is commonly positioned against three families of alternatives. First, compared with Hard Debias, which relies on manually specified gender directions, INLP is data-driven and generalizes beyond word embeddings and binary attributes [2004.07667]. Second, compared with adversarial debiasing, INLP is post-hoc, simple, and interpretable, because it deterministically removes the row space used by a trained linear probe rather than modifying the representation through adversarial dynamics [2004.07667]. Third, unlike naive PCA removal, INLP uses attribute supervision and therefore targets the row space associated with the designated variable rather than unsupervised high-variance directions [2004.07667].

Later extensions preserve the same projection intuition while modifying the representation space or the projector construction. The 2026 kernelized variant for LLM-based recommendation lifts representations with Random Fourier Features, performs INLP in the lifted space, and exports the backbone block of the closed-form projector as the deployed transformation [2603.23780]. The stated motivation is that LLM representations may carry nonlinear attribute signals that ordinary linear INLP can miss [2603.23780].

The acronym also has an unrelated earlier use in signal processing. "Iterative Null-space Projection Method with Adaptive Thresholding in Sparse Signal Recovery and Matrix Completion" [1610.00287] uses the term for a compressed sensing algorithm based on iterative projections of thresholded signals onto the null-space of a sensing matrix. Although it shares the language of null-space projection, that method addresses sparse recovery and matrix completion rather than attribute erasure in learned representations [1610.00287]. The overlap is terminological rather than methodological in the NLP sense.

In current representation analysis, the canonical meaning of INLP refers to the 2020 null-space projection method for removing linear information about a protected or target attribute from embeddings [2004.07667]. Its continuing relevance lies in the combination of closed-form geometry, direct compatibility with linear probing, and post-hoc applicability to arbitrary vector representations, together with a now well-documented need for careful stopping, validation, and control experiments when repeated projections are used for causal interpretation [2506.11673].

Source: https://www.emergentmind.com/topics/iterative-null-space-projection-inlp-cff768dc-8174-44f5-a436-d63426952b3d