Sparse Linear Concept Subspaces (SLiCS)

• SLiCS are representations where concepts are encoded as sparse, linear subspaces within a learned feature space for effective interpretation.
• Different formulations—post hoc discovery, dictionary learning, and training-time anchoring—offer distinct mechanisms for realizing sparse concept geometry.
• SLiCS methods capture multidimensional concept diversity and enable targeted interventions that improve model explanation and retrieval.

Searching arXiv for the cited SLiCS-related papers to ground the article in the current literature. Sparse Linear Concept Subspaces (SLiCS) denotes a class of representations in which concepts are encoded as sparse, linear directions or subspaces inside a learned feature space. In one formulation, a concept is a low-dimensional linear subspace embedded in a hidden feature layer and discovered post hoc from sparse self-expressive structure in neural activations (Vielhaben et al., 2022). In another, an embedding is decomposed into a sparse, non-negative sum of concept-specific components, each lying in a concept-specific positive cone spanned by a learned group of dictionary atoms (Li et al., 27 Aug 2025). A third formulation positions selected concepts during training as axis-aligned directions or axis-aligned subspaces on a unit hypersphere, with geometry chosen to support both reversible steering and permanent removal (Fraser et al., 13 Dec 2025). Across these variants, the unifying idea is that concept structure is linear, sparse, and operationally usable for explanation, retrieval, or control.

1. Formal notion of a concept subspace

In the SSCCD formulation, a concept is a low-dimensional linear subspace $S_k \subset \mathbb{R}^d$ with basis $U_k \in \mathbb{R}^{d \times r_k}$ whose columns are orthonormal and whose intrinsic dimension satisfies $r_k \ll d$. The orthogonal projector onto the concept subspace is $P_{S_k} = U_k U_k^\top$. The relation between a feature vector $x$ and a concept $S_k$ is characterized by principal angles; for a one-dimensional subspace spanned by $x$, the smallest principal angle $\theta_1^{xA}$ measures alignment, with smaller angles indicating stronger concept activation (Vielhaben et al., 2022).

In the supervised dictionary-learning formulation, an embedding $x_\star \in \mathbb{R}^d$ is modeled as

$$\mathbf{x}_\star \approx \sum_{j=1}^{S} \mathbf{v}_j^\star = \sum_{j=1}^{S} \mathbf{B}_j \boldsymbol{\alpha}_j = \mathbf{B}\boldsymbol{\alpha},$$

where each concept-specific component $U_k \in \mathbb{R}^{d \times r_k}$0 lies in a concept-specific positive cone, $U_k \in \mathbb{R}^{d \times r_k}$1, and inactive concepts satisfy $U_k \in \mathbb{R}^{d \times r_k}$2. This makes the support of $U_k \in \mathbb{R}^{d \times r_k}$3 a direct indicator of active concept subspaces. Non-negativity is treated as crucial for interpretability in contrastive spaces, because negative coefficients would represent semantic negation (Li et al., 27 Aug 2025).

In Sparse Concept Anchoring, concepts are either single unit directions $U_k \in \mathbb{R}^{d \times r_k}$4 or axis-aligned subspaces defined by a designated dimension set $U_k \in \mathbb{R}^{d \times r_k}$5. The latent representation is normalized as $U_k \in \mathbb{R}^{d \times r_k}$6, so the geometry lives on the hypersphere. Axis alignment is explicit: a concept may be anchored to one basis direction or confined to a small set of latent coordinates (Fraser et al., 13 Dec 2025).

A recurrent source of confusion is the meaning of sparsity. In SSCCD, sparsity is in the self-expressive representation used for clustering, not necessarily in the basis $U_k \in \mathbb{R}^{d \times r_k}$7 itself. In dictionary-learning SLiCS, sparsity is group-sparse and non-negative at the coefficient level. In Sparse Concept Anchoring, sparsity is induced by allocating only a small number of latent dimensions to a concept and by repulsive regularizers that clear those dimensions for concept occupancy. These are distinct mechanisms that realize the same broad SLiCS idea.

Formulation	Concept geometry	Sparsity mechanism
SSCCD	Low-dimensional linear subspace $U_k \in \mathbb{R}^{d \times r_k}$8	Sparse self-representation $U_k \in \mathbb{R}^{d \times r_k}$9
Dictionary-learning SLiCS	Positive cone spanned by atom group $r_k \ll d$0	Non-negative group-sparse coefficients
Sparse Concept Anchoring	Axis-aligned direction or axis-aligned subspace	Few designated dimensions plus repulsion

2. SSCCD and post hoc concept discovery

Sparse Subspace Clustering for Concept Discovery (SSCCD) begins from hidden feature maps of a trained model. For each input $r_k \ll d$1, the feature map $r_k \ll d$2 is computed at a chosen layer, spatial locations are flattened into feature vectors $r_k \ll d$3, and a random sub-collection is assembled into $r_k \ll d$4. SSCCD then fits a sparse self-representation

$r_k \ll d$5

using the elastic-net SSC objective

$r_k \ll d$6

In practice it uses elastic net subspace clustering (EnSC), with $r_k \ll d$7 in all experiments, $r_k \ll d$8, and an outlier-removal step based on thresholding $r_k \ll d$9 at the $P_{S_k} = U_k U_k^\top$0 percentile before refitting (Vielhaben et al., 2022).

The learned sparse codes define an affinity matrix

$P_{S_k} = U_k U_k^\top$1

Spectral clustering on $P_{S_k} = U_k U_k^\top$2 yields clusters $P_{S_k} = U_k U_k^\top$3, where the number of clusters is either determined from the largest eigengap of the graph Laplacian or set manually; $P_{S_k} = U_k U_k^\top$4 concepts is used in most experiments. For each cluster, SSCCD estimates intrinsic dimension and computes a low-dimensional basis by PCA, with $P_{S_k} = U_k U_k^\top$5 given by the first $P_{S_k} = U_k U_k^\top$6 principal components. The resulting $P_{S_k} = U_k U_k^\top$7 are the concept subspaces.

Application to new samples proceeds by evaluating the smallest principal angle $P_{S_k} = U_k U_k^\top$8 at each spatial location. A concept activation threshold is defined as

$P_{S_k} = U_k U_k^\top$9

and the angle map is upsampled by bilinear interpolation to input resolution. Smaller angles indicate stronger concept activation. Sample-level proximity is defined by $x$0.

Concept relevance is deliberately decoupled from concept discovery. SSCCD thresholds the concept map to obtain a mask $x$1 and then aggregates any attribution method within that mask. With Integrated Gradients, the relevance score is

$x$2

while PredDiff uses inpainting-based perturbations and aggregates attribution over the same mask. The same subspace representation also supports concept-concept similarity through principal angles and Grassmann distance, enabling within-class comparisons, cross-class transfer, and model-to-model transfer (Vielhaben et al., 2022).

This post hoc SLiCS realization is explicitly tied to hidden feature geometry rather than input segmentation. It therefore differs from approaches that define concepts as single directions or depend on model-independent pre-segmentation.

3. Dictionary-learning SLiCS in embedding spaces

The 2025 SLiCS formulation addresses dense vision embeddings such as CLIP, TiTok, and DINOv2 by learning a group-structured dictionary whose groups correspond to concepts. The training data consist of embeddings $x$3 with multi-label annotations $x$4, and the optimization problem is

$x$5

Thus, group activity is consistent with the label structure: inactive concepts are forced to contribute zero (Li et al., 27 Aug 2025).

Coefficient estimation is performed by non-negative least squares with group support constraints. For each sample,

$x$6

subject to $x$7. This enforces group-wise sparsity directly from multi-label supervision. Partial disentanglement for concept-filtered retrieval is defined by projection onto a single concept subspace,

$x$8

which is reported to preserve more information for concept-$x$9 than the joint decomposition when concept subspaces are not orthogonal.

The dictionary is learned by an alternating optimization with guaranteed convergence. Initialization uses per-concept truncated SVD on embeddings from samples carrying the relevant label. Coefficients are updated by NNLS, and atoms are updated by a rank-1 semi-NMF step: a residual matrix $S_k$0 is formed for each atom, rank-1 truncated SVD is computed, and the atom polarity is chosen by a majority rule that maximizes the norm of the non-negative part of the coefficient vector. The paper states that coefficient updates are convex NNLS problems with a unique global minimum per sample, while atom updates can be made conditional to ensure monotone reconstruction-error decrease to a stationary point (Li et al., 27 Aug 2025).

A notable extension is the use of text co-embeddings for semantics. For CLIP, each concept dictionary $S_k$1 can be captioned by retrieving single-word text embeddings that are best reconstructed by $S_k$2 under non-negative reconstruction error. The same co-embedding structure also supports an unsupervised variant: zero-shot cosine scores between image embeddings and concept text embeddings are used to assign pseudo-labels, after which the same supervised SLiCS algorithm is run. This creates a zero-shot–enabled dictionary-learning pipeline without changing the core optimization.

4. Sparse Concept Anchoring and controllable subspaces

Sparse Concept Anchoring (SCA) realizes the SLiCS idea during training rather than after training. The model is an autoencoder with encoder $S_k$3, decoder $S_k$4, latent pre-normalization $S_k$5, normalized latent $S_k$6, and reconstruction $S_k$7. The total loss combines reconstruction, separation, and concept regularization. The separation regularizer is

$S_k$8

with $S_k$9 in experiments, encouraging uniform spread on the hypersphere (Fraser et al., 13 Dec 2025).

Concept organization is controlled by attractive and repulsive regularizers. For a direction concept with unit anchor $x$0, the anchor regularizer

$x$1

attracts labeled examples toward $x$2. For an axis-aligned subspace $x$3, the subspace regularizer

$x$4

penalizes activation outside designated dimensions. Anti-anchor and anti-subspace variants repel samples from the concept direction or subspace, thereby clearing dimensions for later concept occupancy. The paper emphasizes that no $x$5 or group-lasso terms are used; sparsity arises from dimensional confinement and repulsion. Labels are sparse, with “labels for <0.1% of examples per anchored concept” and example counts of “83 ± 8 labeled examples per model” for the red concept and “108 ± 11” for the optional vibrant concept.

The anchored geometry enables two interventions. Reversible suppression removes the concept-aligned component by orthogonal projection. For a single-direction concept,

$x$6

For an axis-aligned subspace $x$7, suppression zeroes the corresponding coordinates:

$x$8

Because $x$9 is unit norm, the suppressed representation generally has $\theta_1^{xA}$0 and is therefore off-manifold.

Permanent removal is implemented by targeted weight ablation. For all dimensions $\theta_1^{xA}$1, the method zeros encoder outputs to those dimensions, sets the corresponding encoder biases to zero, and zeros decoder inputs from those dimensions:

$\theta_1^{xA}$2

These dimensions become dead, so they neither receive nor contribute information. The paper notes that the term “SLiCS” is not explicitly mentioned, but states that SCA realizes the SLiCS idea through sparse, linear concept directions and subspaces, explicit geometric organization, and direct controllability (Fraser et al., 13 Dec 2025).

5. Empirical behavior and application regimes

The empirical behavior of post hoc SLiCS is documented in SSCCD across ImageNet and CelebA. On ImageNet, pretrained VGG16 and ResNet50 are studied on feature layers with comparable spatial sizes, while on CelebA a ResNet50 trained from scratch on 40 attributes achieves “macro AUC 0.92 on test” (Vielhaben et al., 2022). Compared to a PCA/ICE baseline and ACE + TCAV, SSCCD is reported to have a multidimensionality advantage: for high intra-class variance such as ImageNet “basketball,” it yields coherent localized subspaces for “shirt, heads, ball, gym floor/wall,” whereas PCA produces less localized vectors and misses key concepts such as the ball. For low intra-class variance such as “golden retriever,” PCA performs better but still splits coherent structures like the nose into multiple directions, whereas SSCCD captures them as one multidimensional subspace. On “police van,” both VGG16 and ResNet50 discover similar concept sets and relevance rankings, but ResNet50 additionally identifies “emergency lights,” indicating architecture-specific strategy. On CelebA, “Male” and “Female” concepts occupy distinct subspaces even for similar regions such as the nose, and evaluating male concepts on female samples yields “normalized proximity $\theta_1^{xA}$3.” The method’s relevance estimates are also reported to be stable across attribution choices, with “median Spearman rank correlations 0.9–1.0 across 42 ImageNet classes.” In the Smallest Destroying Concepts benchmark, SSCCD shows “the sharpest accuracy decline, outperforming ACE,” while random concept removal yields much weaker declines.

The dictionary-learning SLiCS formulation is evaluated on MIRFlickr25K and MS COCO for concept-filtered retrieval with mAP@20, and on CLIP RN50, CLIP ViT-B/32, TiTok-L-32, and DINOv2 ViT-B/14 embeddings (Li et al., 27 Aug 2025). For CLIP ViT-B/32 on MIRFlickr25K, general retrieval improves from “UF-CLIP=0.728” to “S-SLiCS=0.895,” and sub-label retrieval improves from “0.700” to “0.756.” On MS COCO with the same backbone, general retrieval improves from “0.791” to “0.929±0.002,” and sub-label retrieval from “0.671” to “0.742±0.002.” On DINOv2 ViT-B/14, MIRFlickr25K general retrieval reaches “0.943” versus “UF 0.726,” and MS COCO general retrieval reaches “0.984±0.001” versus “0.793.” The paper also reports that S-SVD is usually second-best, U-SLiCS is third-best, and that U-SLiCS beats UF-CLIP in general retrieval by “0.109–0.134.” These gains are tied to concept-filtered scoring with partial concept components rather than full monolithic embeddings. The same framework supports conditional generation by mapping image-space components into CLIP text space through an orthogonal Procrustes alignment and then using the resulting prompt in image-to-prompt generation.

Sparse Concept Anchoring is evaluated in structured autoencoders. In a 4D suppression model with anchor and subspace regularization, red MSE increases from “0.000646436” to “0.283565700,” while “lime and purple remain at 0.000000000” and magenta increases to “0.048393689” (Fraser et al., 13 Dec 2025). In a 5D weight-ablation model with anchor, anti-subspace, and anti-anchor regularization, red MSE rises from “0.000064271” to “0.343180478,” while cyan remains “essentially unchanged.” The paper relates these outcomes to theoretical error scales: suppression approaches the gray-fallback bound of $\theta_1^{xA}$4, and weight ablation approaches the random on-manifold bound of $\theta_1^{xA}$5. Selectivity is quantified by the relationship between intervention-induced MSE and color similarity, with “R^2 = 0.99” for the selected suppression model and “R^2 = 0.98” for the selected ablation model; across 60 runs, suppression gives “0.95 ± 0.02,” while ablation shows higher variance at “0.86 ± 0.15.”

These results suggest distinct but complementary operating regimes. SSCCD emphasizes localized explanation and transfer in internal feature maps, dictionary-learning SLiCS emphasizes disentanglement of dense embeddings for retrieval and generation, and Sparse Concept Anchoring emphasizes intervention-ready geometry for selective attenuation and removal.

6. Relation to adjacent methods, limitations, and open problems

SLiCS-style approaches are repeatedly contrasted with methods that define concepts as single directions, including TCAV, ACE, ICE, ConceptSHAP, MACE, and PACE. The central claim is not that one-dimensional concept vectors are unusable, but that they often split coherent phenomena into multiple directions and therefore miss intra-concept diversity. SSCCD states that multidimensional subspaces “avoid proliferation of one-dimensional concept vectors that split coherent phenomena,” and that they avoid “model-independent pre-segmentation and heuristic postprocessing” associated with ACE (Vielhaben et al., 2022). The dictionary-learning variant similarly differs from fixed-word sparse coding methods such as SpLiCE by learning concept subspaces from image embeddings, using text only for pseudo-labels and semantic captioning (Li et al., 27 Aug 2025). Sparse Concept Anchoring, in turn, differs from post hoc approaches by prescribing geometry during training rather than discovering it afterward, and it differs from steering-vector methods by attempting to make intervention behavior reliable by design (Fraser et al., 13 Dec 2025).

Several limitations are explicit. SSCCD assumes linear subspaces, so “strongly nonlinear manifolds may be only approximately captured,” and spectral clustering can be expensive for large $\theta_1^{xA}$6 unless approximate methods such as Nyström are used (Vielhaben et al., 2022). The same framework is sensitive to layer choice, feature scaling, cluster fragmentation, and background-dominated clusters. In dictionary-learning SLiCS, performance depends on label quality, dictionary size selection, and the extent of overlap between concept subspaces; the paper notes that too few atoms limit diversity, too many blur concept boundaries, and zero-shot pseudo-labels can be noisy (Li et al., 27 Aug 2025). It also identifies domain shift and scalability of NNLS-plus-SVD updates as practical constraints, and notes that concept visualization for TiTok components remains technically incomplete because decoder-side handling of non-unit tokens requires additional machinery. Sparse Concept Anchoring reports “variability in ablation selectivity across seeds,” the need for careful scheduling of repulsive and attractive terms, the fact that suppression can become off-manifold and rely on decoder fallback biases, and the possibility that fine-tuning could reconstitute a removed behavior after ablation (Fraser et al., 13 Dec 2025). It also states that scaling axis alignment and sparse labeling to transformers and complex domains is an open challenge.

A further misconception is that SLiCS necessarily implies unsupervised discovery. The literature described here shows three distinct regimes: post hoc unsupervised discovery from feature geometry, supervised dictionary learning with multi-label alignment or zero-shot pseudo-labels, and training-time anchoring with minimal supervision. Another misconception is that SLiCS must be axis-aligned. Axis alignment is explicit in Sparse Concept Anchoring, but SSCCD learns arbitrary low-dimensional subspaces, and dictionary-learning SLiCS learns unconstrained atom sets whose positive cones need not align with coordinate axes. This suggests that SLiCS is best understood not as a single algorithm but as a representational principle: concepts are sparse and linear at the level of geometry, while the mechanism that enforces or discovers that geometry can vary substantially across applications.