---
title: Sparse Linear Concept Embeddings (SpLiCE)
url: https://www.emergentmind.com/topics/sparse-linear-concept-embeddings-splice
type: topic
---

# Sparse Linear Concept Embeddings (SpLiCE)

Sparse Linear Concept Embeddings (SpLiCE) is a method for interpreting CLIP representations by expressing a dense image embedding as a sparse, nonnegative linear combination of human-interpretable text concepts. In the 2024 formulation, SpLiCE treats the semantic structure of CLIP’s latent space as a sparse recovery problem, is task-agnostic, can be used without training, and can both explain and replace traditional dense CLIP representations while maintaining high downstream performance and improving interpretability [2402.10376].

## 1. Problem setting and conceptual motivation

Modern vision-language models such as CLIP map images and text into a high-dimensional dense vector space, for example with \(d \approx 512\). Those embeddings support zero-shot classification, retrieval, captioning, and related multimodal applications, but the resulting vectors are not easily interpretable because individual coordinates do not have obvious semantic meaning. This limits their use in settings that require transparency, including bias auditing, model debugging, and transparent decision-making [2402.10376].

SpLiCE addresses the question of whether a CLIP image embedding can be decomposed into a small number of semantically meaningful units. The method assumes that, under mild assumptions, any CLIP image embedding \(z\) can be approximately written as a sparse, nonnegative linear combination of text-based concept embeddings derived from a large vocabulary of single English words. Each nonzero coefficient is then read as indicating how present that concept is in the image. This makes the representation post hoc, sparse, and human-readable in a way that existing alternatives do not provide simultaneously. The problem formulation is explicitly distinct from methods that require concept labels, concept bottlenecks, or qualitative feature visualization [2402.10376].

A central point is that the concepts are not learned from annotated concept data. Instead, they are obtained from CLIP text embeddings of words such as “dog,” “sky,” or “flower,” and the image is represented as a sparse combination over that vocabulary. This makes the method unsupervised in the sense described in the source summary and allows the same representation to be reused across downstream tasks [2402.10376].

## 2. Sparse recovery formulation

Let \(f:\mathbb{R}^{d_i}\to\mathbb{R}^{d}\) be the CLIP image encoder and \(g:\mathbb{R}^{d_t}\to\mathbb{R}^{d}\) the CLIP text encoder. For an image \(x^{img}\in\mathbb{R}^{d_i}\), let \(z=f(x^{img})\) denote its embedding. Let \(\{w_i\}_{i=1\ldots c}\in\mathbb{R}^{d}\) be the CLIP text embeddings of a vocabulary of \(c\) single English words, collected into the matrix
\[
C = [w_1\; w_2\; \ldots\; w_c] \in \mathbb{R}^{d\times c}.
\]
The objective is to find a sparse nonnegative vector \(\alpha\in\mathbb{R}^{c}_{+}\) such that \(C\alpha \approx z\) [2402.10376].

The idealized sparse-recovery problem is written as
\[
\min_{\alpha\in\mathbb{R}^{c}_{+}} \|\alpha\|_0
\qquad
\text{s.t.}\quad
\left\langle z,\frac{C\alpha}{\|C\alpha\|_2}\right\rangle \ge 1-\epsilon,
\]
with an additional sparsity condition \(\|\alpha\|_0 \le k\). The formulation is motivated by the fact that \(C\) is overcomplete, with \(c \gg d\), so direct projection is not available and sparsity is imposed so that only \(k \ll c\) concepts explain \(z\) [2402.10376].

Because that problem is combinatorial, SpLiCE uses the usual \(\ell_1\) relaxation and replaces the cosine constraint with an \(\ell_2\) reconstruction term:
\[
\min_{w\in\mathbb{R}^{c}_{+}} \|Cw-z\|_2^2 + 2\lambda \|w\|_1.
\]
Here \(\lambda>0\) trades off reconstruction accuracy against sparsity. Empirically, setting \(\lambda\) so that \(\|w\|_0 \approx 10\)–\(30\) yields highly interpretable sparse codes with minimal performance loss [2402.10376].

The nonnegativity constraint is structurally important. It restricts the decomposition to additive concept presence rather than allowing arbitrary positive and negative cancellations among concept embeddings. In the interpretation presented in the source material, the nonzero entries of the solution are literally English words weighted by prominence in the image [2402.10376].

## 3. Computational pipeline

SpLiCE is implemented as a precomputation stage followed by a test-time sparse coding stage. The method first computes and stores an image-cone mean
\[
\mu_{img} = \mathbb{E}_{x\in MSCOCO}[f(x)],
\]
a concept-cone mean
\[
\mu_{con} = \frac{1}{c}\sum_{i=1}^{c} g(word_i),
\]
and a centered, unit-normalized concept matrix
\[
C = [\,\mathrm{normalize}(g(word_i)-\mu_{con})\,]_{i=1\ldots c}\in\mathbb{R}^{d\times c}.
\]
These steps define the dictionary over which sparse concept recovery is performed [2402.10376].

At test time, for a new image \(x^{img}\), the algorithm computes \(z_{raw}=f(x^{img})\), then mean-centers and renormalizes it as
\[
z=\mathrm{normalize}(z_{raw}-\mu_{img}).
\]
It then solves the nonnegative Lasso
\[
w^* = \arg\min_{w\ge 0} \|Cw-z\|_2^2 + 2\lambda \|w\|_1.
\]
The reported implementation uses scikit-learn’s Lasso with `positive=True`, choosing \(\lambda\) so that the solution has \(k \approx 15\)–\(30\) nonzeros. Coordinate descent or ADMM are given as example solvers [2402.10376].

SpLiCE can also reconstruct a dense approximation from the sparse code. The reconstruction is
\[
\hat z = \mathrm{normalize}(Cw^*),
\qquad
\hat z_{final} = \mathrm{normalize}(\hat z + \mu_{img}).
\]
This reconstructed vector is intended to approximate the original CLIP embedding while remaining linked to a sparse set of explicit concepts. The method therefore produces two complementary objects: a sparse concept-weight vector \(w^*\), which is directly interpretable, and a reconstructed dense embedding \(\hat z_{final}\), which is compatible with existing CLIP-based downstream pipelines [2402.10376].

## 4. Downstream use as a replacement for dense CLIP embeddings

Because \(\hat z_{final} \approx z\), SpLiCE is designed so that the reconstructed vector can be substituted directly for the original CLIP embedding in downstream pipelines. The paper reports results for zero-shot classification, linear probing, and retrieval, with the central pattern that a relatively small number of active concepts preserves most of the utility of the original dense representation [2402.10376].

For zero-shot classification using the prompt “A photo of a \(\langle class\rangle\),” the reported behavior depends on the dataset. On CIFAR-100, with only \(k=5\) concepts, SpLiCE retains \(\approx 90\%\) of CLIP’s zero-shot accuracy, and by \(k \approx 20\) it matches CLIP. On MIT-States, similar trends are reported, with \(k \approx 10\) sufficing. On ImageNet, the task is described as more challenging because of species names, but performance recovers if the ImageNet labels themselves are added into the concept vocabulary [2402.10376].

For linear probes, probes trained on dense CLIP and evaluated on SpLiCE embeddings with \(k \approx 23\) lose less than \(1\%\) accuracy on CIFAR-100. Retraining a probe directly on SpLiCE embeddings also yields \(\approx\) CLIP-level accuracy. For text-image retrieval on MSCOCO, at \(k \approx 50\) nonzeros, text-to-image recall@5 drops by only \(\sim 15\%\) relative to dense CLIP [2402.10376].

These results suggest a specific interpretation of the reconstruction-sparsity trade-off: sparse concept decompositions can act not only as explanatory overlays on top of CLIP, but also as operational surrogates for dense embeddings in pipelines that were not designed for interpretability. The reported evidence does not imply lossless replacement in every setting, but it does indicate that the performance cost can remain small over a range of tasks and sparsity levels [2402.10376].

## 5. Interpretability use cases

The main benefit of SpLiCE is that the nonzero entries of \(w^*\) are English words with explicit weights. The paper presents several case studies in which this structure is used to inspect dataset bias, intervene on model behavior, and monitor shifts in data distributions [2402.10376].

In a spurious-correlation analysis of CIFAR100 man and woman classes, decomposing images labeled “woman” reveals that “bra” and “swimwear” appear in the top 10 concepts. The summary identifies this as signaling a gender stereotype bias in the dataset. A histogram of \(w_{\text{“swimwear”}}\) shows that the concept is activated far more in the “woman” class than in “man,” providing a direct concept-level diagnostic rather than a latent-space heuristic [2402.10376].

A model-editing example is given on CelebA for the concept “glasses.” A zero-shot gender classifier is trained on SpLiCE embeddings. Baseline zero-shot accuracy on “glasses” presence is reported as \(88\%\). The intervention then zeros out the coordinate corresponding to “glasses” before reconstructing \(\hat z\). After intervention, the same zero-shot classifier’s accuracy on detecting glasses falls to \(69\%\), compared with \(50\%\) random, while gender-prediction accuracy remains \(\approx 97\%\). The table excerpt is reproduced below [2402.10376].

| Method | Gender | Glasses |
|---|---:|---:|
| ZS-CLIP (dense) | 0.98 | 0.91 |
| ZS-SpLiCE (before) | 0.97 | 0.88 |
| ZS-SpLiCE (after) | 0.96 | 0.69 |
| Probe (dense) | 0.89 | 0.88 |
| Probe (after) | 0.85 | 0.59 |

A third case study concerns distribution-shift monitoring in Stanford Cars ’91–’12. Images are decomposed year by year and the mean concept weights for “convertible” and “yellow” are plotted against the actual fraction of cars with those attributes in each year. The reported result is that the SpLiCE concept weights closely track the ground-truth prevalence, suggesting a lightweight method for detecting temporal shifts in large image streams [2402.10376].

## 6. Limitations, misconceptions, and related formulations

The limitations reported for the 2024 method are concrete. SpLiCE uses only single-word English concepts; multi-word phrases or compositional concepts such as “butter chicken” and “fire truck” are out of scope unless they are manually added. The decomposition is purely linear, and richer basis expansions, including shallow networks over \(C\), are proposed as possible alternatives. The method uses an \(\ell_1\) relaxation of \(\ell_0\), and other sparse-coding formulations such as greedy OMP, hard \(\ell_0\) projections, or binary weights are explicitly identified as possible directions for exploration. Concept vocabulary choice also matters: using the LAION top-10k tokens is described as working well, while domain-specific vocabularies or learned dictionaries can alter performance and interpretability [2402.10376].

Several common misconceptions are therefore ruled out by the reported formulation. SpLiCE is not a method for recovering arbitrary compositional language semantics from CLIP, because its vocabulary is restricted. It is not an inherently nonlinear concept model, because the decomposition is linear. It is also not a concept-bottleneck model that requires concept supervision, since the 2024 method is post hoc, unsupervised in the sense of requiring no labeled concept data, and usable without retraining CLIP [2402.10376].

The name “SpLiCE” also appears in adjacent but non-identical lines of work, which creates a terminological ambiguity. In NLP, Templeton’s sparse-coding method transforms pretrained dense word embeddings into sparse embeddings in which each dimension corresponds to a natural language word or a specific grammatical concept, with a fixed basis and Lasso-style inference [2004.13847]. In interpretable image classification, “Sparse Linear Concept Discovery Models” proposes a CLIP-based framework with a single sparse linear layer and Bernoulli concept indicators inferred through a variational objective [2308.10782]. Related post hoc disentanglement work includes CEDAR, which learns an invertible transformation with a top-\(k\) sparsity bottleneck without increasing dimensionality [2605.22679], and SLiCS, which uses supervised dictionary learning with sparse, non-negative combinations of concept-specific groups of atoms [2508.20322]. In representation steering for language models, SSAE instead sparse-autoencodes embedding differences and proves permutation-scaling identifiability under a sparse-shifts model [2502.12179]. Earlier concept-embedding work on bag-of-concepts densification learned continuous concept vectors and efficient vector aggregation rather than CLIP-based sparse concept decompositions [1702.03342].

Taken together, these related formulations indicate that “Sparse Linear Concept Embeddings” names a broader family resemblance rather than a single standardized architecture. In the specific sense established by the CLIP interpretation paper, however, SpLiCE denotes a sparse, nonnegative, post hoc decomposition of CLIP embeddings into human-interpretable text concepts, with downstream substitution, bias auditing, and intervention as its principal demonstrated use cases [2402.10376].

Source: https://www.emergentmind.com/topics/sparse-linear-concept-embeddings-splice