LEACE: Least-Squares Concept Erasure

• LEACE is a post-hoc method that transforms learned representations to make target concept information undetectable to any linear classifier.
• It employs a closed-form affine transformation that minimizes expected squared perturbation, ensuring minimal distortion of non-target features.
• LEACE is pivotal in fairness and interpretability research, serving as a canonical baseline for linear concept erasure techniques.

Searching arXiv for LEACE and closely related papers to ground the article in the cited literature. Least-Squares Concept Erasure (LEACE) is a post-hoc method for editing a representation $X \in \mathbb{R}^d$ so that a target concept $Z$ becomes undetectable to every linear classifier, while the edited representation remains as close as possible to the original. The method is defined in closed form, guarantees perfect linear concept erasure, and minimizes expected squared perturbation over a broad class of quadratic norms. It has been motivated by both fairness, where $Z$ may be a protected attribute such as gender or race, and interpretability, where erasure is used as an intervention on internal representations to test whether a model uses a concept in downstream computation (Belrose et al., 2023). Subsequent work has treated LEACE as the canonical linear baseline for higher-order erasure methods, characterizing it as the method that removes “first-order information” by making all classes have the same mean while distorting the data only as little as is necessary (2502.02820).

1. Problem formulation and scope

LEACE operates on a learned representation $X$ and a concept variable $Z$, typically one-hot labels in a multiclass setting but also, in the original formulation, potentially continuous targets under ordinary least squares regression (Belrose et al., 2023). The erasure map is affine,

$r(x)=\mathbf P x + \mathbf b,$

and is chosen so that the transformed representation $r(X)$ no longer contains usable information about $Z$ for any linear predictor.

The central design objective combines two desiderata. First, the target concept should be erased for the entire family of linear probes, not merely for one trained classifier. Second, the intervention should be surgical rather than destructive: everything unrelated to the concept should be preserved as much as possible. In the original paper, this is formalized as a constrained mean-squared perturbation problem over affine maps, which is why the method is called Least-Squares Concept Erasure (Belrose et al., 2023).

For one-hot labels $Z$, the original paper states that the following are equivalent: $X$ linearly guards $Z$0; the trivially attainable loss is optimal for all convex losses; all class-conditional means are equal to the global mean; and the cross-covariance vanishes,

$Z$1

This equivalence makes LEACE a second-moment method: perfect linear erasure is achieved by eliminating the cross-covariance structure through which linear predictors access the concept (Belrose et al., 2023).

A later polynomial-predictor treatment frames LEACE as the $Z$2 case of a broader moment-matching principle. In that formulation, if

$Z$3

then no degree-1 polynomial predictor, hence no linear predictor, can improve on the best constant predictor under convex loss. That paper therefore treats LEACE as the closed-form linear concept erasure method that removes all linearly available information while minimally distorting the representation (2502.02820).

2. Linear guardedness and statistical characterization

The original theory gives an exact characterization of when an affine transformation erases a concept for all linear classifiers. If $Z$4, then for an affine eraser $Z$5, the transformed representation $Z$6 linearly guards $Z$7 if and only if

$Z$8

Equivalently,

$Z$9

The eraser must therefore annihilate the entire subspace of feature directions correlated with the concept (Belrose et al., 2023).

This condition is both necessary and sufficient. In the classification setting with one-hot $Z$0, zero cross-covariance is equivalent to equality of class-conditional means. Consequently, after LEACE, every linear predictor

$Z$1

is no better, for every convex loss, than the best constant predictor. The guarantee is hypothesis-class complete within the linear setting: it is not a statement about one probe failing, but about the impossibility of linear recovery altogether (Belrose et al., 2023).

The 2025 polynomial extension makes this same point from a different angle. Its sufficiency theorem states that if the class-conditional moments up to order $Z$2 equal the unconditional moments, then the trivially attainable loss cannot be improved upon for degree-$Z$3 polynomial predictors under convex loss. Specializing to $Z$4 recovers LEACE’s condition. Its necessity theorem, which the paper notes applies to categorical cross-entropy, shows that if the constant predictor minimizes the expected loss among degree-$Z$5 polynomial predictors, then the corresponding class-conditional moments must match the unconditional ones. For LEACE, this again reduces to equality of class-conditional means, so LEACE removes exactly the information needed to defeat all linear predictors (2502.02820).

3. Closed-form operator and geometric interpretation

In the centered case, LEACE is constructed from the covariance of the representation and the representation–concept cross-covariance. Let

$Z$6

and define the whitening map

$Z$7

where $Z$8 denotes the Moore–Penrose pseudoinverse. The whitened cross-covariance is

$Z$9

and the orthogonal projector onto its column space is

$X$0

The least-squares-optimal erasing linear map is then

$X$1

For general $X$2, the optimal bias is

$X$3

so the final affine map is

$X$4

This is the main closed-form formula of the method (Belrose et al., 2023).

The geometry is covariance-aware. The procedure centers $X$5, whitens it on the support of $X$6, projects onto the whitened concept subspace, unwhitens that component, and subtracts it. The use of pseudoinverses is essential because $X$7 may be singular; hidden representations can live in a lower-dimensional subspace, so whitening is defined only on the support of the covariance (Belrose et al., 2023).

A notable feature is that the optimal map is generally not an orthogonal projection in the original feature space. It is an oblique projection: $X$8 This matters because previous approaches often treated orthogonal projection as the natural notion of minimal distortion. LEACE shows that the true least-squares optimum need not be orthogonal: the eraser can reconstruct some erased directions from concept-free correlated structure in the remaining features, thereby preserving more information than an orthogonal nullspace projection (Belrose et al., 2023).

The dimension of the removed concept subspace is

$X$9

which is often much smaller than $Z$0. Computationally, LEACE requires only estimates of $Z$1, $Z$2, and $Z$3, followed by matrix square roots, pseudoinverses, and a projector construction. The required covariance statistics can be computed in streaming fashion, which the original paper highlights as useful for language-model hidden states (Belrose et al., 2023).

4. Optimality and relation to neighboring methods

Among all affine maps satisfying the exact erasure constraint $Z$4, LEACE minimizes

$Z$5

for every positive semidefinite inner-product norm induced by $Z$6. The paper explicitly includes the Euclidean norm $Z$7 and the Mahalanobis norm $Z$8. This is a strong minimal-distortion theorem: LEACE is not merely feasible, but the unique most surgical eraser up to zero-variance or zero-weight degeneracies (Belrose et al., 2023).

This optimality distinguishes LEACE from several earlier linear erasure families. INLP iteratively trains linear predictors and projects onto their null spaces; the LEACE paper argues that this can remove many more dimensions than necessary and cause collateral damage. RLACE, by contrast, formulates linear erasure as an adversarial optimization game and can come close empirically, but it requires gradient-based minimax optimization. LEACE is closed-form, much faster, and does not require iterative training (Belrose et al., 2023).

The relationship to the precursor “Linear Adversarial Concept Erasure” is especially close in the least-squares setting. Ravfogel et al. derive an exact regression solution

$Z$9

which removes the empirical covariance direction $r(x)=\mathbf P x + \mathbf b,$0 and makes the optimal linear regressor powerless. That result is best understood as a close conceptual precursor: it identifies cross-covariance as the object to erase, but it constrains the erasure to an orthogonal projection and does not provide LEACE’s general affine residualization operator or its simultaneous minimal-distortion guarantee across inner-product-induced norms (Ravfogel et al., 2022).

The LEACE paper also situates itself relative to SAL, mean projection, and fair PCA. These methods can achieve linear guardedness under appropriate conditions because they remove the column space of $r(x)=\mathbf P x + \mathbf b,$1 or equalize class means, but they are generally not least-squares optimal. In the binary case, mean projection removes the centroid-difference direction; full-rank SAL removes the span of the left singular vectors of $r(x)=\mathbf P x + \mathbf b,$2. LEACE differs by exploiting the covariance geometry of $r(x)=\mathbf P x + \mathbf b,$3, which is why its optimum is typically oblique rather than orthogonal (Belrose et al., 2023).

5. Applications and empirical behavior

The original paper develops LEACE as a practical intervention method for representation analysis and debiasing. In BERT embeddings on Bias in Bios, LEACE is reported as the only method to achieve random-accuracy gender prediction with a small edit. A profession classifier trained on projected representations has profession accuracy $r(x)=\mathbf P x + \mathbf b,$4 after LEACE versus $r(x)=\mathbf P x + \mathbf b,$5 on original representations; the TPR gap drops from $r(x)=\mathbf P x + \mathbf b,$6 to $r(x)=\mathbf P x + \mathbf b,$7; and the correlation between profession-specific TPR gap and the proportion of women in the profession drops from $r(x)=\mathbf P x + \mathbf b,$8 before erasure to $r(x)=\mathbf P x + \mathbf b,$9 after erasure. Retraining the downstream classifier recovers some accuracy, to $r(X)$0, but also increases the TPR gap to $r(X)$1, indicating that downstream learning can reintroduce bias via remaining proxies or nonlinear pathways (Belrose et al., 2023).

For interpretability, the paper introduces “concept scrubbing,” which erases a concept from every intermediate representation in a deep network by sequential fitting. Sequential fitting is necessary because earlier interventions change the distribution seen by later layers. The procedure was applied to transformer hidden states at the input of each transformer block immediately after normalization. In single-layer amnesic probing on BERT, INLP requires 20 iterations and still does not fully erase part-of-speech information, while LEACE reduces representation rank by only 17 dimensions, compared with 360 for INLP. In autoregressive LLMs such as Pythia and LLaMA, scrubbing linearly available part-of-speech information causes a large increase in perplexity in bits per UTF-8 byte, while removing a random subspace of the same rank has little effect; SAL causes even larger degradation, which the paper interprets as collateral damage rather than stronger causal evidence (Belrose et al., 2023).

A subsequent study on slowing learning by erasing low-order statistics uses LEACE as the canonical linear baseline and applies it directly to image datasets including CIFAR-10, CIFARNet, and SVHN. Learning speed is measured with prequential minimum description length, described there as equivalent to the area under the learning curve. Across MLPs, parameter-matched LeNets, Swin Transformer V2, and ConvNeXt V2, LEACE “consistently increases MDL and final loss across all models and tasks.” The effect is reported as roughly equal between architectures and as varying little across widths and depths; increasing width beyond roughly 500 or 1K neurons does not materially undo the slowdown. That paper therefore concludes that LEACE is “a reliable method for making features less salient and more difficult to learn” (2502.02820).

6. Extensions, limits, and stronger notions of erasure

LEACE’s guarantee is exact but narrow: it concerns linear detectability, not mutual information in the unrestricted information-theoretic sense. The original paper explicitly cautions that nonlinear classifiers may still recover information about $r(X)$2 from $r(X)$3, and that distribution shift can break the empirical zero-covariance condition learned from finite samples. It also warns against overinterpreting concept erasure as a full causal decomposition: intervention-based changes in behavior are informative, but they do not constitute a complete causal semantics of the model (Belrose et al., 2023).

The 2025 quadratic extension makes this limitation precise by distinguishing linearly or quadratically available information from Shannon information. In that framework, LEACE removes first-order information by making all classes have the same mean, while QLEACE additionally equalizes class covariances so that all quadratically available information is removed. Empirically, however, QLEACE can inject higher-order Shannon information because it uses class-dependent affine transports; sophisticated architectures can exploit that injected structure, and the method can backfire. LEACE is presented there as more reliable precisely because it stays within the linear setting and avoids this class-dependent information injection (2502.02820).

A broader information-theoretic treatment draws an even sharper distinction. “Fundamental Limits of Perfect Concept Erasure” defines perfect erasure as

$r(X)$4

that is, statistical independence between the erased representation and the concept. Within that stronger framework, LEACE is described as a linear concept erasure method that makes concept groups share the same centroid representation and thereby prevents linear adversaries from extracting concept information, while still allowing stronger nonlinear adversaries to recover it. The paper’s toy example explicitly illustrates that after LEACE, groups can cease to be linearly separable while remaining nonlinearly separable (Chowdhury et al., 25 Mar 2025).

Kernel-based nonlinear extensions reinforce the same point from another direction. “Kernelized Concept Erasure” shows that concept removal can be carried out in an RKHS so as to neutralize a chosen nonlinear adversary class, but the protection does not transfer reliably to different kernels or to MLP adversaries. This suggests that exhaustive nonlinear concept erasure is substantially harder than linear erasure and does not admit a straightforward analogue of LEACE’s exact, closed-form universality within the linear hypothesis class (Ravfogel et al., 2022).

In that sense, LEACE occupies a specific and well-defined position in the concept-erasure literature. It does not solve perfect information-theoretic sanitization, nor does it exhaust nonlinear encodings. Its contribution is more precise: it shows that perfect linear concept erasure has an exact covariance characterization, an explicit closed-form solution, and a strong minimal-distortion theorem, making it the standard reference point for linear concept erasure and for subsequent attempts to generalize concept removal beyond first-order structure (Belrose et al., 2023).