Conceptor Steering Matrix

• Conceptor steering matrices are linear operators computed from data that use regularized identity mappings for selective control in embedding or activation spaces.
• They enable Boolean-like operations (AND, OR, NOT) to flexibly combine steering objectives and filter specific activation directions in neural networks.
• Applications include refining word embeddings, steering LLM activations for improved performance, and mitigating catastrophic forgetting in continual learning.

A conceptor steering matrix is a linear operator derived from data, used to selectively amplify, suppress, or isolate specific directions in embedding or activation space. It arises from the matrix conceptor formalism—originally defined as regularized identity mappings—and has been adapted for both post-processing pre-trained word vectors and for in-context steering of deep neural network activations. Unlike scalar or simple vector shift steering, the conceptor steering matrix admits a biologically interpretable parametrization (aperture), supports Boolean-like algebraic operations, and offers precise, structured control over both semantic and functional properties in LLMs and continual learning systems.

1. Mathematical Foundations of Conceptor Matrices

Let $x \in \mathbb R^n$ be a data vector (e.g., word embedding or network activation), with empirical covariance $R = \mathbb E[x x^\top]\in\mathbb R^{n\times n}$. The conceptor matrix $C\in\mathbb R^{n\times n}$ is obtained as the minimizer of a regularized auto-encoding objective:

$$J(C) = \mathbb E\big[\|x - C x\|_2^2\big] + \alpha^{-2} \|C\|_F^2$$

where $\|\cdot\|_F$ denotes Frobenius norm and $\alpha>0$ is the aperture. Solving yields

$C = R\,(R + \alpha^{-2} I)^{-1}$

The eigenvalues $\sigma_i(C) = t_i / (t_i+\alpha^{-2})$ (where $t_i$ are the eigenvalues of $R$) interpolate smoothly between zero (as $\alpha\to 0$) and one (as $\alpha \to \infty$), resulting in a soft, data-driven projection. The conceptor steering matrix is typically realized as the negated conceptor:

$\neg C = I - C$

which suppresses components aligned with the principal subspace of $R$ while retaining the complement (Liu et al., 2018, Postmus et al., 9 Oct 2024).

2. Steering Operations and Boolean Algebra

Conceptors admit Boolean-like operations—OR ($\vee$), AND ($\wedge$), and NOT ($\neg$)—that correspond to algebraic manipulations of the underlying covariance structure:

• OR: $$C_1 \vee C_2 = (R_1 + R_2) (R_1 + R_2 + \alpha^{-2}I)^{-1}$$
• AND: $C_1 \wedge C_2 = (C_1^{-1} + C_2^{-1} - I)^{-1}$
• NOT: $\neg C = I - C$

These operations enable the composition of multiple steering objectives (e.g., enforcing “topic=A AND NOT style=B”), granting unprecedented flexibility compared to traditional vector-based steering or hard subspace projections. The algebraic structure underpins recent steering methodologies in both static word embeddings and residual activations of LLMs (Postmus et al., 9 Oct 2024, Apolinario et al., 21 Nov 2024).

3. Algorithms and Hyperparameters

Practical construction of a conceptor steering matrix proceeds as follows:

Step	Description	Mathematical Formulation
Covariance Estimation	Stack $m$ vectors as columns of $X\in\mathbb R^{n\times m}$	$R = (1/m)\, X X^\top$
Conceptor Computation	Compute $C$ via closed form	$C = R (R + \alpha^{-2} I)^{-1}$
Steering Matrix	Obtain negated conceptor	$\neg C = I - C$
Application	To activation $h$, apply soft projection (possibly scaled)	$h' = \beta_c\, \neg C\, h$ or $h' = \beta_c\, C\, h$
Hyperparameters	Aperture $\alpha$, scaling $\beta_c$	$\alpha\in[10^{-3}, 1]$, $\beta_c$ typically grid-searched in $[0.5, 5]$

In continual learning, conceptors are computed per-task, per-layer, and updated cumulatively via the OR operation. The steering matrix is used as a gradient projector: $$\nabla_{W^\ell}\mathcal L \leftarrow (I-C_{<t}^\ell)\nabla_{W^\ell}\mathcal L$$ (Apolinario et al., 21 Nov 2024). Stability and transfer are governed by eigenstructure and the aperture parameter, which mediates the degree of filtering.

4. Applications in Embeddings and Model Steering

Word Embedding Post-processing

Conceptor steering matrices suppress high-variance (often “spurious”) directions in pre-trained word vectors, resulting in improved semantic similarity scores (e.g., SimLex-999) and downstream task performance (e.g., dialogue state tracking) (Liu et al., 2018). The operation yields more isotropic, semantically salient embedding geometries.

LLMs

In LLMs, conceptor steering is used for activation engineering—inducing stylistic, functional, or factual patterns in outputs by steering hidden states at inference. Empirical studies demonstrate superior zero-shot control compared to additive steering, especially when combining multiple objectives via AND/OR operations (Postmus et al., 9 Oct 2024). The matrix form allows layer-wise fusion, efficiency optimizations, and fine-grained subspace control.

Continual Learning

Conceptor-based gradient projection (e.g., as in CODE-CL) steers learning updates into subspaces orthogonal, or “pseudo-orthogonal” via soft complements, to those relevant to previously learned tasks. This mitigates catastrophic forgetting while supporting forward transfer in highly correlated multitask settings (Apolinario et al., 21 Nov 2024).

Multi-concept Steering with Sparse Autoencoding

Recent advances leverage sparse autoencoders to recover identifiable, disentangled directions of concept shift, yielding a family of “Conceptor Steering Matrices” $M_c = W_d E_{cc} W_e$ that isolate and manipulate single interpretable concepts in modern LLM embeddings. This facilitates unsupervised, stable, and functionally precise steering (Joshi et al., 14 Feb 2025).

5. Theoretical and Empirical Properties

Conceptor steering matrices provide soft, data-driven projections regularized by their aperture, leading to several distinct properties:

• Continuity: Eigenvalues in $(0,1)$ ensure no direction is nullified absolutely unless $\alpha\to\infty$.
• Boolean flexibility: Complex composite steering operators are constructed without recomputing high-dimensional covariances.
• Interpretability: In the sparse autoencoding regime, provable identification (up to scaling and permutation) of concept vectors is attainable under mild conditions of richness and linearity in the embedding map (Joshi et al., 14 Feb 2025).
• Empirical Robustness: Performance on function-steering and semantic tasks is robust to moderate $\alpha$ and number of samples; best results are typically found at midrange aperture settings (Postmus et al., 9 Oct 2024).
• Efficiency: The major computational cost is offline matrix inversion; at inference, matrix–vector multiplication is the dominant operation, which can be subsumed into subsequent model layers.

6. Implementation Considerations

Implementation concerns include sample-efficiency, numerical rank, and choice of basis:

• When $m<n$ or $R$ is ill-conditioned, regularization (e.g., $R\leftarrow R+\varepsilon I$) stabilizes inversion.
• On high-dimensional data, low-rank approximations (e.g., restricting to top principal components) are viable.
• For continual learning, the pseudo-inverse is needed if conceptors are rank-deficient.
• Pipelines in recent works use grid searches for $\alpha$ and scaling $\beta_c$, mean-centering for anisotropic activations, and recommend assembling complex steering matrices by chained Boolean operations rather than data concatenation (Postmus et al., 9 Oct 2024, Apolinario et al., 21 Nov 2024).

7. Empirical Impact and Outlook

The conceptor steering matrix paradigm has unified a family of steering, filtering, and projection-based interventions:

• In word vectors, it achieves up to 5–10 point improvements on standard semantic similarity tasks and consistent gains in language understanding (Liu et al., 2018).
• In LLM activation engineering, conceptor matrices outperform vector-based steering by 10–30 percentage points absolute in function transfer accuracy, and their Boolean algebra admits combinatorial objective composition (Postmus et al., 9 Oct 2024).
• In continual learning, conceptor-based projection offers finer granularity than hard orthogonalization, enabling both stability and transfer with nuanced trade-offs (Apolinario et al., 21 Nov 2024).
• Unsupervised conceptor-like mappings from sparse autoencoders support provable identifiability and disentangled multi-concept control in modern embeddings (Joshi et al., 14 Feb 2025).

The expressivity, interpretability, and compositional capacity of conceptor steering matrices position them as fundamental tools for structured representation manipulation, robust continual learning, and scalable activation engineering in modern NLP and deep learning systems.