Subspace Separation & Causal Projection

• The paper introduces subspace separation as decomposing high-dimensional data into orthogonal subspaces that isolate causal processes from noise and confounding features.
• Causal projection employs projection operators, SVD/PCA, and difference-in-means to extract intervention-ready causal components from complex signals.
• Methodologies enhance system identification in neuroscience, safe generative modeling in AI, and fairness in Bayesian networks via rigorous statistical validation.

Subspace separation and causal projection are methodologies for decomposing complex high-dimensional data, latent representations, and neural or generative model activations into explicit subspaces that isolate causal processes from confounding, noise, or non-causal features. These techniques underlie advances in neuroscience system identification, causal analysis in time series, safe generative modeling, LLM behavior intervention, and formal causal inference in Bayesian networks. The central mathematical principle is to identify low-dimensional subspaces in which directed (causal) relationships, robust interventions, and effective attribution can be precisely quantified and manipulated via orthogonal projection operators, singular-value decompositions, or structured system identification.

1. Mathematical Foundations of Subspace Separation and Projection

Subspace separation leverages linear algebraic decomposition to partition an ambient vector space (e.g., $\mathbb{R}^d$ of neural activations, text embeddings, or time-series states) into orthogonal subspaces corresponding to different functional, behavioral, or causal roles. The generic construction is a separation function

$$f_{\mathrm{sep}}(z) = (z_{\mathrm{co}}, z_{\mathrm{de}}) = (M_{\mathrm{co}} z, M_{\mathrm{de}} z),$$

with $M_{\mathrm{de}}$, $M_{\mathrm{co}}$ projection matrices such that $M_{\mathrm{de}} + M_{\mathrm{co}} = I$, $M_{\mathrm{co}} M_{\mathrm{de}}^T = 0$. The operator $P$ defined by $P(z) = M_{\mathrm{de}} z$ projects any signal onto the causal/deconfounder subspace $Z_{\mathrm{de}}$, while $(I-P)$ produces the confounder/non-causal residual. Such decompositions can be constructed via difference-in-means (DiffMean), SVD/PCA, or learned mutual information maximization, depending on context (Amin, 17 May 2025, Vennemeyer et al., 25 Sep 2025, Chen et al., 21 Mar 2025).

In generative and identification contexts, the system (states, outputs) are embedded in a state-space representation, with latent/causal neuronal states $x_k$ evolving via a block-structured state-transition matrix $\hat{A}$, and observations $y_k$ as convolutions of these states with region-specific basis functions (Bakhtiari et al., 2011). In time-series source separation, an instantaneous mixing process $x(t) = A s(t) + n(t)$ produces observed signals, with hidden sources $s(t)$ initially inseparable except for constraints imposed by autocovariance and time-varying variance structure (Zhang et al., 2012).

2. Subspace Identification Methodologies

Subspace identification algorithms estimate causal system matrices (connectivity or transition coefficients) from observational data by algebraically extracting the dominant subspace of future-to-past correlations. For brain connectivity recovery, embeddings of measured BOLD signals yield block-Hankel matrices $(Y_p, Y_f)$; projecting future on past and performing SVD recovers the principal observability subspace, from which least-squares system matrices $\hat{A}$ and $C$ are computed (Bakhtiari et al., 2011).

In source separation for EEG/MEG, blockwise likelihoods leveraging AR models with time-varying variance drive natural gradient updates to unmixing matrices, while local autocovariances are computed via blockwise empirical correlations. Ultimately, the estimated sources are separated with precise envelopes $\sigma_i^2(t)$ for each component (Zhang et al., 2012).

In LLMs, difference-in-means directions are calculated per behavior and dataset, SVD yields subspaces wherein specific behaviors reside, and projection operators $P_U$ or $P_{\perp U}$ enable targeted interventions, ablations, or validation of independence properties (Vennemeyer et al., 25 Sep 2025).

For causal inference in Bayesian networks, Attribution Projection Calculus (AP-Calculus) provides formal existence, uniqueness, and optimality theorems for the deconfounder subspace, defined via mutual information maximization between source features and label outcomes (Amin, 17 May 2025).

3. Causal Projection: Theory and Implementation

Causal projection is the act of mapping an arbitrary signal, activation, or embedding onto a subspace representing a specific causal factor, while removing all orthogonal confounding components. Mathematically, given a concept direction $u_c$, the causal projection onto the complementary subspace is $(I - u_c u_c^T) e$, which nullifies the concept strength $\alpha = u_c^T e$ in any prompt embedding or activation vector (Chen et al., 21 Mar 2025).

In brain networks, causal projection after subspace identification isolates the M$\times$M top block of $\hat{A}$, directly quantifying the effective connectivity (EC) between neuronal regions, with statistical validation via surrogate data shuffling. Only statistically significant nonzero off-diagonal entries are retained as directed causal edges (Bakhtiari et al., 2011).

For MEG/EEG, projection of separated sources into envelope space and fitting a multivariate GARCH model identifies directed “causality-in-variance”—nonzero $\gamma_{ij}$ coefficient signals that envelope variance in source $j$ drives changes in source $i$. These connections map directly to effective connectivity graphs (Zhang et al., 2012).

In LLMs, projection and subspace-removal operations empirically demonstrate that distinct behaviors (e.g., sycophantic agreement, genuine agreement, sycophantic praise) can be independently isolated, amplified, or suppressed via subspace-based activation intervention, validating a strong linear causal geometry (Vennemeyer et al., 25 Sep 2025).

4. Applications Across Modalities

The subspace separation and projection framework generalizes across domains:

Domain	Subspace Target	Causal Projection Role
fMRI/EEG/MEG	Latent neuronal states, envelopes	EC graph estimation, high-order modulator
Diffusion models (SAFER)	Concept direction in text embedding	Erasure of style/object, robust content control
LLMs	Behavior-specific rows/subspaces	Independent steering and decontamination
Bayesian networks (AP-Calculus)	Deconfounder subspace	Optimal causal attribution, fairness, uncertainty

In generative modeling, SAFER identifies and projects out unwanted style/object concepts using text embeddings. Textual inversion and SVD/PCA yield precise low-dimensional concept subspaces, which are then orthogonally projected out to causally erase generation capabilities for that concept, robustly across synonym and variant prompts, validated quantitatively across artistic style, object, and nudity scenarios (Chen et al., 21 Mar 2025).

LLMs are steered at inference by intervention on identified subspace directions. Selectivity metrics (ratio of targeted change to cross-behavior leakage) confirm that behaviors are linearly and causally encoded, with little off-target impact (Vennemeyer et al., 25 Sep 2025).

In neuroscience, subspace identification followed by causal projection yields effective connectivity diagrams for complex resting-state or task-driven brain networks, robust against noise, signal variability, and sampling artifacts (Bakhtiari et al., 2011, Zhang et al., 2012).

AP-Calculus expands causal mediation analysis in supervised learning: separating intermediate representation into causal and spurious subspaces enables explicit attribution, regularization against spurious correlation, fairness auditing, and principled uncertainty quantification (Amin, 17 May 2025).

5. Statistical Validation, Robustness, and Identifiability

Statistical validation in subspace causal modeling is critical. Significance of estimated connections is independently verified:

• Surrogate generation via phase shuffling destroys coupling while preserving spectra; thresholding of test statistics $S_{ij}$ (based on difference to surrogate mean and standard deviation) yields robust EC graphs (Bakhtiari et al., 2011).
• In source separation, ARCH-LM and $\ell_1$ penalized GARCH provide envelope nonstationarity tests and sparse edge selection (Zhang et al., 2012).
• In concept erasure, ablated subspace projections (text-only, TI-only, TI+expansion) are directly compared for completeness and robustness; only expansion achieves full removal under all tested prompt varieties (Chen et al., 21 Mar 2025).
• LLM behavior selectivity is measured by AUROC, behavior rate change, and cross-intervention leakage. Subspace-removal controls empirically demonstrate independence of underlying representations (Vennemeyer et al., 25 Sep 2025).
• In AP-Calculus, theoretical optimality guarantees that causal projection never increases error and strictly improves prediction under presence of confounding (Amin, 17 May 2025).

Identifiability depends on specific conditions: full-rank signal mixing, sufficient nonstationarity or variation among sources, distinct autocorrelation or envelope variance structure, or mutual information maximization yielding unique solutions (Zhang et al., 2012, Bakhtiari et al., 2011, Amin, 17 May 2025).

6. Theoretical Significance and Extensions

Subspace causal projection formalizes and extends the classical do-calculus. In AP-Calculus, causal interventions correspond to projecting onto the deconfounder subspace and integrating out the confounder complement, subsuming insertion–deletion, action-observation, and marginalization rules as algebraic identities (Amin, 17 May 2025). This framework generalizes to high-dimensional supervised models, LLMs, and generative models, supplying a coherent calculus for attribution, intervention, and correction.

In neuroscience, subspace-based methods circumvent the limitations imposed by indirect observation, noise, and hemodynamic filtering by projecting inference onto latent neuronal states and their causal transitions (Bakhtiari et al., 2011). Source separation plus higher-order variance modeling resolves connectivity and mediatory dynamics not captured by standard ICA or FC analyses (Zhang et al., 2012).

In contemporary generative and inference architectures, causal subspace projection offers robust tools for targeted removals, fairness auditing, uncertainty management, and behavior steering, unifying disparate strands of causal analysis under a shared algebraic principle.

7. Summary Table: Core Constructs

Construct	Definition/Method	Primary Role
Separation function	$$f_{\mathrm{sep}}(z) = (M_{\mathrm{co}}z, M_{\mathrm{de}}z)$$	Orthogonal partitioning
Causal projection $P$	$P(z) = M_{\mathrm{de}}z$	Extraction of causal component
SVD/PCA-based subspaces	Principal directions in sample covariance	Data-driven basis estimation
State-space embedding	Structured latent node stacking in system ID	Neuronal EC recovery
Activation addition	$h' = h + \alpha w_b$	Direct behavior steering
GARCH envelope model	Multivariate variance propagation in AR sources	Higher-order causal links

These methods provide a rigorous foundation for subspace separation and causal projection, advancing the resolution and intervention capacity of modern data-driven causal inference across neuroscience, generative AI, and supervised machine learning (Bakhtiari et al., 2011, Zhang et al., 2012, Chen et al., 21 Mar 2025, Vennemeyer et al., 25 Sep 2025, Amin, 17 May 2025).