---
title: 'TextCAM: Enhancing CAM with Text'
url: https://www.emergentmind.com/topics/textcam
type: topic
---

# TextCAM: Enhancing CAM with Text

Searching arXiv for the specified paper and closely related background works on CAM and CLIP.
2510.01004 TextCAM Explaining Class Activation Map with Text
2508.12345 Grad-CAM revisited
2103.00020 CLIP-style vision-language models
1704.04595 Grad-CAM
1512.04150 CAM classification activation mapping
TextCAM is an explanation framework for deep vision models that augments Class Activation Mapping (CAM) with natural-language descriptions, with the goal of specifying not only where a model attends but also what visual attributes likely support its decision [2510.01004]. In the formulation reported for the method, standard CAM and its variants provide spatially localized heat maps but offer little semantic insight into whether a highlighted region is being used for color, shape, texture, or related attributes. TextCAM addresses this limitation by combining the precise spatial localization of CAM with the semantic alignment of vision-language models, specifically by deriving channel-level semantic representations using CLIP embeddings and linear discriminant analysis (LDA), aggregating these semantics with CAM weights, and retrieving sparse textual attributes from a candidate phrase vocabulary [2510.01004]. The framework is also extended to partition feature channels into semantically coherent groups, producing multiple phrase-labeled heat maps for finer-grained visual-textual explanations.

## 1. Conceptual basis and motivation

Standard CAM methods are described as producing a heat map
\[
V_c(x)\;=\;\sum_{j=1}^d w_j^c\,A_j(x),
\]
which indicates spatial support for class \(c\) but does not identify the semantic attribute underlying that support [2510.01004]. The reported limitation is not merely qualitative. A highlighted region may correspond to a bird’s beak, yet a human observer cannot determine whether the classifier relies on its color, shape, pattern, or glossiness. The paper therefore frames TextCAM as a response to three specific deficiencies of spatial-only explanations: absence of semantic resolution, reduced capacity to detect spurious cues such as background color, and limited ability to summarize fine-grained evidence in human-interpretable terms [2510.01004].

TextCAM is positioned as an enrichment layer over CAM rather than a replacement for CAM. In the reported formulation, it preserves the CAM-style localization signal while attaching natural-language descriptors to the underlying channel activity. This suggests a two-level explanation regime: a spatial level inherited from CAM and an attribute level induced through CLIP-space semantics. A plausible implication is that the method is intended for settings in which localization alone is insufficient for interpretability diagnostics, especially in fine-grained recognition and bias analysis.

## 2. Channel-level semantic representations

The first technical component of TextCAM constructs a semantic vector \(s_j\) for each activation channel \(j\) at a chosen layer [2510.01004]. Let the network produce activation maps \(\{A_j(x)\}_{j=1}^d\). For each channel, TextCAM computes global-average pooled activations
\[
a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},
\]
then collects the top-\(M\) images by \(a_j\) as the positive set \(\mathcal{P}_j\) and the bottom-\(M\) images as the negative set \(\mathcal{N}_j\), yielding \(2M\) labeled examples with \(y_i\in\{+1,-1\}\) [2510.01004]. The positive/negative split is therefore channel-specific and is defined by empirical activation strength.

Each example \(x_i\) is embedded by CLIP’s image encoder into \(\phi(x_i)\in\mathbb{R}^D\). The method then computes class means
\[
\mu_{+} \;=\;\frac1{|\mathcal{P}_j|}\sum_{i\in\mathcal{P}_j}\phi(x_i),\quad
\mu_{-} \;=\;\frac1{|\mathcal{N}_j|}\sum_{i\in\mathcal{N}_j}\phi(x_i),
\]
together with within-class and between-class scatter matrices
\[
S_W \;=\;\sum_{i\in\mathcal{P}_j}(\phi(x_i)-\mu_{+})(\phi(x_i)-\mu_{+})^T
       \;+\;\sum_{i\in\mathcal{N}_j}(\phi(x_i)-\mu_{-})(\phi(x_i)-\mu_{-})^T,
\]
\[
S_B \;=\;(\mu_{+}-\mu_{-})(\mu_{+}-\mu_{-})^T.
\]
LDA is then used to find a projection direction \(p_j\) maximizing class separation:
\[
p_j \;=\;\arg\max_{p}\;\frac{p^T\,S_B\,p}{p^T\,S_W\,p}
\;\longleftrightarrow\;
S_B\,p = \lambda\,S_W\,p.
\]
This direction is weighted by the channel’s average activation to obtain the final channel semantic representation:
\[
s_j \;=\;a_j\,p_j.
\]
By construction, each \(s_j\) lies in the same \(D\)-dimensional CLIP space as text embeddings [2510.01004].

The significance of this step is that a feature channel is not treated as an opaque basis element. Instead, its semantics are estimated through discriminative separation in CLIP space between images that strongly activate the channel and images that weakly activate it. This suggests that TextCAM interprets channels relationally: a channel “means” what distinguishes its highly activating examples from its weakly activating examples in a shared multimodal embedding space.

## 3. Aggregation with CAM weights and sparse phrase retrieval

For a test image \(x\), standard CAM provides class-specific channel weights \(w_j^c\). TextCAM forms a single image-semantic vector
\[
T_c(x)
\;=\;
\sum_{j=1}^d
w^c_j\,s_j(x)
\;\in\;\mathbb{R}^D
\]
by aggregating channel semantics with the same weights that drive the CAM explanation [2510.01004]. The paper notes that one may optionally \(\ell_2\)-normalize each \(s_j\) or normalize \(T_c\) before retrieval, but reports that direct summation suffices in practice.

Textual explanation is then posed as sparse selection over a vocabulary of candidate phrases \(\{t_1,\dots,t_N\}\) with CLIP text embeddings \(E=[e_1,\dots,e_N]\in\mathbb{R}^{D\times N}\). TextCAM solves for nonnegative phrase weights \(\omega\in\mathbb{R}^N_{\ge0}\):
\[
\omega^*
=\arg\min_{\omega\ge0}
\frac12\big\|T_c - E\,\omega\big\|_2^2
\;+\;\alpha\|\omega\|_1
\;+\;\beta\,\omega^T\bigl((\mathbf1\mathbf1^T-I)\circ E^TE\bigr)\,\omega.
\]
The objective contains three distinct terms. The reconstruction term aligns the retrieved textual mixture with the aggregated image-semantic vector. The \(L_1\) term enforces sparsity. The covariance penalty
\[
(\mathbf1\mathbf1^T-I)\circ E^TE
\]
discourages selecting highly correlated phrases. The resulting small quadratic program is solved via ADMM, and the top-\(K\) nonzero entries of \(\omega^*\) are taken as the final textual attributes [2510.01004].

This retrieval stage formalizes explanation as controlled concept selection rather than nearest-neighbor captioning. A plausible implication is that TextCAM aims to avoid redundant attribute lists by jointly encouraging semantic fit, sparsity, and diversity.

## 4. Phrase-conditioned saliency groups

TextCAM further extends from a single pooled explanation to multiple phrase-labeled heat maps by clustering channels into \(K\) groups \(G_1,\dots,G_K\) [2510.01004]. Let the selected phrase embeddings be \(\{e_1,\dots,e_K\}\), and define weighted channel semantics \(\bar s_j = w_j^c\,s_j\). The grouping objective is
\[
\{g_j\}
=
\arg\min_{g_j\in\{1\ldots K\}}
\sum_{k=1}^K
n_k\,\bigl\|\mu_k - e_k\bigr\|^2,
\]
where \(G_k=\{j:g_j=k\}\), \(n_k=|G_k|\), and
\[
\mu_k=\frac1{n_k}\sum_{j\in G_k}\bar s_j.
\]
The optimization is reported as NP-hard, so the paper adopts a classical greedy relocation local-search procedure [2510.01004].

The procedure is described in four stages. First, initialize each channel assignment by nearest phrase embedding:
\[
g_j\leftarrow\arg\min_k\|\bar s_j - e_k\|.
\]
Second, compute group sums \(S_k\) and sizes \(n_k\). Third, iterate over channels in random order; for a channel currently in group \(a\), evaluate the objective change \(\Delta_{a\to b}\) for every target group \(b\neq a\), reassigning if any move decreases the objective and updating \(S_a,S_b,n_a,n_b\). Fourth, repeat sweeps until no reassignment improves the objective or a maximum number of sweeps is reached, with the paper giving “e.g. 5 000” as a limit [2510.01004].

Once the grouping is fixed, the method renders phrase-specific heat maps
\[
V_c^k(x) \;=\;\sum_{j\in G_k} w_j^c\,A_j(x)
\]
and labels each map with phrase \(t_k\) [2510.01004]. This converts a monolithic saliency map into a set of semantically partitioned saliency components. In the paper’s examples, this enables explanations such as separate maps for “long-necked,” “gangly,” and “slender beak” in an ImageNet “American egret” prediction, or separate part-aligned phrases such as “pointed beak,” “yellow belly,” and “spotted wings” in CUB-200 [2510.01004].

## 5. Empirical evaluations and reported results

The paper reports experiments on ImageNet, CLEVR, CUB, and DomainNet, and characterizes the resulting explanations as faithful and interpretable while preserving model fidelity [2510.01004]. The evaluation spans qualitative analysis, human interpretability assessment, controlled diagnostic tests, fine-grained attribute agreement, and a debiasing use case.

| Setting | Reported result | Context |
|---|---|---|
| ImageNet-1K | User understanding improved + 37% | Human study on interpretability |
| CLEVR Diagnostic | Top-1 textual accuracy \(\mathrm{Acc}^{\mathrm{TXT}}=100\%\) | Both frozen heads over 100 test images |
| CUB-200 | 86% of generated attributes matched ground-truth part-attribute annotations | Fine-grained textual agreement |
| CUB-200 | Classification accuracy unchanged (\(\pm 0.2\%\)) | After adding TextCAM hooks |
| CLEVR shape debiasing | 75.67% \(\rightarrow\) 84.33% (+8.66 pp) | After ablating color-dominant channels |

On ImageNet-1K, the reported models are ResNet-50 and Swin, and the CAM families examined include Grad-CAM, Layer-CAM, Score-CAM, Eigen-CAM, and Finer-CAM [2510.01004]. The qualitative claim is that TextCAM consistently yields plausible phrases faithful to the underlying CAM regions. The human study reports that TextCAM rationales improved user understanding by \(+37\%\).

On the CLEVR diagnostic, the evaluation uses two frozen heads, \(M_a\) for shape and \(M_b\) for color, with held-out distractors described as yellow cylinders [2510.01004]. For a red ball, the shape model produces phrases such as “ball” and “round,” whereas the color model produces phrases such as “red” and “warm-colored,” despite the same spatial focus. The reported Top-1 textual accuracy is \(\mathrm{Acc}^{\mathrm{TXT}}=100\%\) for both heads over 100 test images. The paper also states that the spatial maps are insensitive to distractors, so the textual attribution is what reflects the correct factorized property.

On CUB-200, the paper reports that \(86\%\) of generated attributes matched ground-truth part-attribute annotations and that classification accuracy was unchanged within \(\pm 0.2\%\) after adding TextCAM hooks [2510.01004]. This is presented as evidence that the explanation mechanism does not materially perturb the base classifier while still recovering fine-grained descriptive content.

On DomainNet, ResNet-50 is trained on four domains—real, clipart, sketch, and quickdraw—and the ratio of color/shape/texture phrases in Top-1 TextCAM results is computed [2510.01004]. The reported trend is high shape reliance in sketch and quickdraw and high color reliance in real and clipart, matching domain priors.

## 6. Interpretive uses, diagnostics, and scope

A central claim of TextCAM is that textualized saliency supports interpretability diagnostics that are difficult to perform with spatial maps alone [2510.01004]. The paper explicitly identifies detection of spurious correlations as one such use case, since a conventional heat map may reveal that the model attends to a region without revealing whether the operative cue is the object itself, background color, or another attribute. By attaching attributes to CAM responses, TextCAM provides a mechanism for identifying which factor is being used within the highlighted region.

The reported debiasing experiment on a CLEVR shape classifier illustrates this use. Under color-biased data, original shape accuracy is reported as \(75.67\%\). After ablating color-dominant channels mined through TextCAM analysis, accuracy increases to \(84.33\%\), a gain of \(+8.66\) percentage points, and the intervention is described as requiring no further training [2510.01004]. The paper interprets this as confirmation that TextCAM can guide targeted feature-space interventions.

The method is also stated to preserve model fidelity. On CUB-200, classification accuracy remains unchanged within \(\pm 0.2\%\) after adding TextCAM hooks [2510.01004]. In this sense, TextCAM is presented as an explanatory overlay rather than a retrained surrogate. The paper further summarizes the framework as augmenting any CAM variant by lifting channel activations into a CLIP-LDA semantic space, aggregating them via CAM weights, and retrieving sparse, diverse textual concepts, optionally grouped for fine-grained heat maps, while requiring no modification to the original vision model [2510.01004].

A plausible implication is that TextCAM occupies an intermediate position between post hoc saliency analysis and concept-based explanation. It does not replace the underlying visual evidence map, but it does convert that evidence into a vocabulary-aligned representation that can be audited, compared across domains, and used for channel-level interventions. Within the scope reported by the paper, its primary contribution is therefore not new localization but semantic resolution of existing CAM explanations.

Source: https://www.emergentmind.com/topics/textcam