Sparse Linear Concept Embeddings (SpLiCE)

• Sparse Linear Concept Embeddings (SpLiCE) is a method that represents dense CLIP embeddings as a sparse, nonnegative combination of human-interpretable text concepts.
• It formulates the decomposition as a sparse recovery problem using an ℓ1 relaxation, ensuring that only a few key concepts explain the image embedding.
• SpLiCE not only improves interpretability for tasks like bias auditing and model debugging but also maintains high performance in downstream applications.

Sparse Linear Concept Embeddings (SpLiCE) is a method for interpreting CLIP representations by expressing a dense image embedding as a sparse, nonnegative linear combination of human-interpretable text concepts. In the 2024 formulation, SpLiCE treats the semantic structure of CLIP’s latent space as a sparse recovery problem, is task-agnostic, can be used without training, and can both explain and replace traditional dense CLIP representations while maintaining high downstream performance and improving interpretability (Bhalla et al., 2024).

1. Problem setting and conceptual motivation

Modern vision-LLMs such as CLIP map images and text into a high-dimensional dense vector space, for example with $d \approx 512$. Those embeddings support zero-shot classification, retrieval, captioning, and related multimodal applications, but the resulting vectors are not easily interpretable because individual coordinates do not have obvious semantic meaning. This limits their use in settings that require transparency, including bias auditing, model debugging, and transparent decision-making (Bhalla et al., 2024).

SpLiCE addresses the question of whether a CLIP image embedding can be decomposed into a small number of semantically meaningful units. The method assumes that, under mild assumptions, any CLIP image embedding $z$ can be approximately written as a sparse, nonnegative linear combination of text-based concept embeddings derived from a large vocabulary of single English words. Each nonzero coefficient is then read as indicating how present that concept is in the image. This makes the representation post hoc, sparse, and human-readable in a way that existing alternatives do not provide simultaneously. The problem formulation is explicitly distinct from methods that require concept labels, concept bottlenecks, or qualitative feature visualization (Bhalla et al., 2024).

A central point is that the concepts are not learned from annotated concept data. Instead, they are obtained from CLIP text embeddings of words such as “dog,” “sky,” or “flower,” and the image is represented as a sparse combination over that vocabulary. This makes the method unsupervised in the sense described in the source summary and allows the same representation to be reused across downstream tasks (Bhalla et al., 2024).

2. Sparse recovery formulation

Let $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$ be the CLIP image encoder and $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$ the CLIP text encoder. For an image $x^{img}\in\mathbb{R}^{d_i}$, let $z=f(x^{img})$ denote its embedding. Let $\{w_i\}_{i=1\ldots c}\in\mathbb{R}^{d}$ be the CLIP text embeddings of a vocabulary of $c$ single English words, collected into the matrix

$$C = [w_1\; w_2\; \ldots\; w_c] \in \mathbb{R}^{d\times c}.$$

The objective is to find a sparse nonnegative vector $\alpha\in\mathbb{R}^{c}_{+}$ such that $z$0 (Bhalla et al., 2024).

The idealized sparse-recovery problem is written as

$z$1

with an additional sparsity condition $z$2. The formulation is motivated by the fact that $z$3 is overcomplete, with $z$4, so direct projection is not available and sparsity is imposed so that only $z$5 concepts explain $z$6 (Bhalla et al., 2024).

Because that problem is combinatorial, SpLiCE uses the usual $z$7 relaxation and replaces the cosine constraint with an $z$8 reconstruction term: $z$9 Here $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$0 trades off reconstruction accuracy against sparsity. Empirically, setting $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$1 so that $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$2–$f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$3 yields highly interpretable sparse codes with minimal performance loss (Bhalla et al., 2024).

The nonnegativity constraint is structurally important. It restricts the decomposition to additive concept presence rather than allowing arbitrary positive and negative cancellations among concept embeddings. In the interpretation presented in the source material, the nonzero entries of the solution are literally English words weighted by prominence in the image (Bhalla et al., 2024).

3. Computational pipeline

SpLiCE is implemented as a precomputation stage followed by a test-time sparse coding stage. The method first computes and stores an image-cone mean

$f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$4

a concept-cone mean

$f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$5

and a centered, unit-normalized concept matrix

$f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$6

These steps define the dictionary over which sparse concept recovery is performed (Bhalla et al., 2024).

At test time, for a new image $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$7, the algorithm computes $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$8, then mean-centers and renormalizes it as

$f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}$9

It then solves the nonnegative Lasso

$g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$0

The reported implementation uses scikit-learn’s Lasso with positive=True, choosing $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$1 so that the solution has $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$2–$g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$3 nonzeros. Coordinate descent or ADMM are given as example solvers (Bhalla et al., 2024).

SpLiCE can also reconstruct a dense approximation from the sparse code. The reconstruction is

$g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$4

This reconstructed vector is intended to approximate the original CLIP embedding while remaining linked to a sparse set of explicit concepts. The method therefore produces two complementary objects: a sparse concept-weight vector $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$5, which is directly interpretable, and a reconstructed dense embedding $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$6, which is compatible with existing CLIP-based downstream pipelines (Bhalla et al., 2024).

4. Downstream use as a replacement for dense CLIP embeddings

Because $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$7, SpLiCE is designed so that the reconstructed vector can be substituted directly for the original CLIP embedding in downstream pipelines. The paper reports results for zero-shot classification, linear probing, and retrieval, with the central pattern that a relatively small number of active concepts preserves most of the utility of the original dense representation (Bhalla et al., 2024).

For zero-shot classification using the prompt “A photo of a $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$8,” the reported behavior depends on the dataset. On CIFAR-100, with only $g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}$9 concepts, SpLiCE retains $x^{img}\in\mathbb{R}^{d_i}$0 of CLIP’s zero-shot accuracy, and by $x^{img}\in\mathbb{R}^{d_i}$1 it matches CLIP. On MIT-States, similar trends are reported, with $x^{img}\in\mathbb{R}^{d_i}$2 sufficing. On ImageNet, the task is described as more challenging because of species names, but performance recovers if the ImageNet labels themselves are added into the concept vocabulary (Bhalla et al., 2024).

For linear probes, probes trained on dense CLIP and evaluated on SpLiCE embeddings with $x^{img}\in\mathbb{R}^{d_i}$3 lose less than $x^{img}\in\mathbb{R}^{d_i}$4 accuracy on CIFAR-100. Retraining a probe directly on SpLiCE embeddings also yields $x^{img}\in\mathbb{R}^{d_i}$5 CLIP-level accuracy. For text-image retrieval on MSCOCO, at $x^{img}\in\mathbb{R}^{d_i}$6 nonzeros, text-to-image recall@5 drops by only $x^{img}\in\mathbb{R}^{d_i}$7 relative to dense CLIP (Bhalla et al., 2024).

These results suggest a specific interpretation of the reconstruction-sparsity trade-off: sparse concept decompositions can act not only as explanatory overlays on top of CLIP, but also as operational surrogates for dense embeddings in pipelines that were not designed for interpretability. The reported evidence does not imply lossless replacement in every setting, but it does indicate that the performance cost can remain small over a range of tasks and sparsity levels (Bhalla et al., 2024).

5. Interpretability use cases

The main benefit of SpLiCE is that the nonzero entries of $x^{img}\in\mathbb{R}^{d_i}$8 are English words with explicit weights. The paper presents several case studies in which this structure is used to inspect dataset bias, intervene on model behavior, and monitor shifts in data distributions (Bhalla et al., 2024).

In a spurious-correlation analysis of CIFAR100 man and woman classes, decomposing images labeled “woman” reveals that “bra” and “swimwear” appear in the top 10 concepts. The summary identifies this as signaling a gender stereotype bias in the dataset. A histogram of $x^{img}\in\mathbb{R}^{d_i}$9 shows that the concept is activated far more in the “woman” class than in “man,” providing a direct concept-level diagnostic rather than a latent-space heuristic (Bhalla et al., 2024).

A model-editing example is given on CelebA for the concept “glasses.” A zero-shot gender classifier is trained on SpLiCE embeddings. Baseline zero-shot accuracy on “glasses” presence is reported as $z=f(x^{img})$0. The intervention then zeros out the coordinate corresponding to “glasses” before reconstructing $z=f(x^{img})$1. After intervention, the same zero-shot classifier’s accuracy on detecting glasses falls to $z=f(x^{img})$2, compared with $z=f(x^{img})$3 random, while gender-prediction accuracy remains $z=f(x^{img})$4. The table excerpt is reproduced below (Bhalla et al., 2024).

Method	Gender	Glasses
ZS-CLIP (dense)	0.98	0.91
ZS-SpLiCE (before)	0.97	0.88
ZS-SpLiCE (after)	0.96	0.69
Probe (dense)	0.89	0.88
Probe (after)	0.85	0.59

A third case study concerns distribution-shift monitoring in Stanford Cars ’91–’12. Images are decomposed year by year and the mean concept weights for “convertible” and “yellow” are plotted against the actual fraction of cars with those attributes in each year. The reported result is that the SpLiCE concept weights closely track the ground-truth prevalence, suggesting a lightweight method for detecting temporal shifts in large image streams (Bhalla et al., 2024).

6. Limitations, misconceptions, and related formulations

The limitations reported for the 2024 method are concrete. SpLiCE uses only single-word English concepts; multi-word phrases or compositional concepts such as “butter chicken” and “fire truck” are out of scope unless they are manually added. The decomposition is purely linear, and richer basis expansions, including shallow networks over $z=f(x^{img})$5, are proposed as possible alternatives. The method uses an $z=f(x^{img})$6 relaxation of $z=f(x^{img})$7, and other sparse-coding formulations such as greedy OMP, hard $z=f(x^{img})$8 projections, or binary weights are explicitly identified as possible directions for exploration. Concept vocabulary choice also matters: using the LAION top-10k tokens is described as working well, while domain-specific vocabularies or learned dictionaries can alter performance and interpretability (Bhalla et al., 2024).

Several common misconceptions are therefore ruled out by the reported formulation. SpLiCE is not a method for recovering arbitrary compositional language semantics from CLIP, because its vocabulary is restricted. It is not an inherently nonlinear concept model, because the decomposition is linear. It is also not a concept-bottleneck model that requires concept supervision, since the 2024 method is post hoc, unsupervised in the sense of requiring no labeled concept data, and usable without retraining CLIP (Bhalla et al., 2024).

The name “SpLiCE” also appears in adjacent but non-identical lines of work, which creates a terminological ambiguity. In NLP, Templeton’s sparse-coding method transforms pretrained dense word embeddings into sparse embeddings in which each dimension corresponds to a natural language word or a specific grammatical concept, with a fixed basis and Lasso-style inference (Templeton, 2020). In interpretable image classification, “Sparse Linear Concept Discovery Models” proposes a CLIP-based framework with a single sparse linear layer and Bernoulli concept indicators inferred through a variational objective (Panousis et al., 2023). Related post hoc disentanglement work includes CEDAR, which learns an invertible transformation with a top-$z=f(x^{img})$9 sparsity bottleneck without increasing dimensionality (Kubaty et al., 21 May 2026), and SLiCS, which uses supervised dictionary learning with sparse, non-negative combinations of concept-specific groups of atoms (Li et al., 27 Aug 2025). In representation steering for LLMs, SSAE instead sparse-autoencodes embedding differences and proves permutation-scaling identifiability under a sparse-shifts model (Joshi et al., 14 Feb 2025). Earlier concept-embedding work on bag-of-concepts densification learned continuous concept vectors and efficient vector aggregation rather than CLIP-based sparse concept decompositions (Shalaby et al., 2017).

Taken together, these related formulations indicate that “Sparse Linear Concept Embeddings” names a broader family resemblance rather than a single standardized architecture. In the specific sense established by the CLIP interpretation paper, however, SpLiCE denotes a sparse, nonnegative, post hoc decomposition of CLIP embeddings into human-interpretable text concepts, with downstream substitution, bias auditing, and intervention as its principal demonstrated use cases (Bhalla et al., 2024).