TextCAM: Enhancing CAM with Text

• TextCAM is an explanation framework that augments CAM by integrating CLIP embeddings with LDA for channel-level semantic representations.
• It aggregates sparse textual attributes with spatial heat maps to provide dual insights into where and why a model focuses during classification.
• Empirical evaluations on benchmarks like ImageNet and CUB-200 demonstrate improved interpretability and effective bias diagnostics without impacting accuracy.

Searching arXiv for the specified paper and closely related background works on CAM and CLIP. (Zhao et al., 1 Oct 2025) TextCAM Explaining Class Activation Map with Text (Jockar et al., 17 Aug 2025) Grad-CAM revisited (Radford et al., 2021) CLIP-style vision-LLMs (You et al., 2017) Grad-CAM (Zhou et al., 2015) CAM classification activation mapping TextCAM is an explanation framework for deep vision models that augments Class Activation Mapping (CAM) with natural-language descriptions, with the goal of specifying not only where a model attends but also what visual attributes likely support its decision (Zhao et al., 1 Oct 2025). In the formulation reported for the method, standard CAM and its variants provide spatially localized heat maps but offer little semantic insight into whether a highlighted region is being used for color, shape, texture, or related attributes. TextCAM addresses this limitation by combining the precise spatial localization of CAM with the semantic alignment of vision-LLMs, specifically by deriving channel-level semantic representations using CLIP embeddings and linear discriminant analysis (LDA), aggregating these semantics with CAM weights, and retrieving sparse textual attributes from a candidate phrase vocabulary (Zhao et al., 1 Oct 2025). The framework is also extended to partition feature channels into semantically coherent groups, producing multiple phrase-labeled heat maps for finer-grained visual-textual explanations.

1. Conceptual basis and motivation

Standard CAM methods are described as producing a heat map

$V_c(x)\;=\;\sum_{j=1}^d w_j^c\,A_j(x),$

which indicates spatial support for class $c$ but does not identify the semantic attribute underlying that support (Zhao et al., 1 Oct 2025). The reported limitation is not merely qualitative. A highlighted region may correspond to a bird’s beak, yet a human observer cannot determine whether the classifier relies on its color, shape, pattern, or glossiness. The paper therefore frames TextCAM as a response to three specific deficiencies of spatial-only explanations: absence of semantic resolution, reduced capacity to detect spurious cues such as background color, and limited ability to summarize fine-grained evidence in human-interpretable terms (Zhao et al., 1 Oct 2025).

TextCAM is positioned as an enrichment layer over CAM rather than a replacement for CAM. In the reported formulation, it preserves the CAM-style localization signal while attaching natural-language descriptors to the underlying channel activity. This suggests a two-level explanation regime: a spatial level inherited from CAM and an attribute level induced through CLIP-space semantics. A plausible implication is that the method is intended for settings in which localization alone is insufficient for interpretability diagnostics, especially in fine-grained recognition and bias analysis.

2. Channel-level semantic representations

The first technical component of TextCAM constructs a semantic vector $s_j$ for each activation channel $j$ at a chosen layer (Zhao et al., 1 Oct 2025). Let the network produce activation maps $\{A_j(x)\}_{j=1}^d$. For each channel, TextCAM computes global-average pooled activations

$a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$

then collects the top-$M$ images by $a_j$ as the positive set $\mathcal{P}_j$ and the bottom-$M$ images as the negative set $c$0, yielding $c$1 labeled examples with $c$2 (Zhao et al., 1 Oct 2025). The positive/negative split is therefore channel-specific and is defined by empirical activation strength.

Each example $c$3 is embedded by CLIP’s image encoder into $c$4. The method then computes class means

$c$5

together with within-class and between-class scatter matrices

$c$6

$c$7

LDA is then used to find a projection direction $c$8 maximizing class separation: $c$9 This direction is weighted by the channel’s average activation to obtain the final channel semantic representation: $s_j$0 By construction, each $s_j$1 lies in the same $s_j$2-dimensional CLIP space as text embeddings (Zhao et al., 1 Oct 2025).

The significance of this step is that a feature channel is not treated as an opaque basis element. Instead, its semantics are estimated through discriminative separation in CLIP space between images that strongly activate the channel and images that weakly activate it. This suggests that TextCAM interprets channels relationally: a channel “means” what distinguishes its highly activating examples from its weakly activating examples in a shared multimodal embedding space.

3. Aggregation with CAM weights and sparse phrase retrieval

For a test image $s_j$3, standard CAM provides class-specific channel weights $s_j$4. TextCAM forms a single image-semantic vector

$s_j$5

by aggregating channel semantics with the same weights that drive the CAM explanation (Zhao et al., 1 Oct 2025). The paper notes that one may optionally $s_j$6-normalize each $s_j$7 or normalize $s_j$8 before retrieval, but reports that direct summation suffices in practice.

Textual explanation is then posed as sparse selection over a vocabulary of candidate phrases $s_j$9 with CLIP text embeddings $j$0. TextCAM solves for nonnegative phrase weights $j$1: $j$2 The objective contains three distinct terms. The reconstruction term aligns the retrieved textual mixture with the aggregated image-semantic vector. The $j$3 term enforces sparsity. The covariance penalty

$j$4

discourages selecting highly correlated phrases. The resulting small quadratic program is solved via ADMM, and the top-$j$5 nonzero entries of $j$6 are taken as the final textual attributes (Zhao et al., 1 Oct 2025).

This retrieval stage formalizes explanation as controlled concept selection rather than nearest-neighbor captioning. A plausible implication is that TextCAM aims to avoid redundant attribute lists by jointly encouraging semantic fit, sparsity, and diversity.

4. Phrase-conditioned saliency groups

TextCAM further extends from a single pooled explanation to multiple phrase-labeled heat maps by clustering channels into $j$7 groups $j$8 (Zhao et al., 1 Oct 2025). Let the selected phrase embeddings be $j$9, and define weighted channel semantics $\{A_j(x)\}_{j=1}^d$0. The grouping objective is

$\{A_j(x)\}_{j=1}^d$1

where $\{A_j(x)\}_{j=1}^d$2, $\{A_j(x)\}_{j=1}^d$3, and

$\{A_j(x)\}_{j=1}^d$4

The optimization is reported as NP-hard, so the paper adopts a classical greedy relocation local-search procedure (Zhao et al., 1 Oct 2025).

The procedure is described in four stages. First, initialize each channel assignment by nearest phrase embedding: $\{A_j(x)\}_{j=1}^d$5 Second, compute group sums $\{A_j(x)\}_{j=1}^d$6 and sizes $\{A_j(x)\}_{j=1}^d$7. Third, iterate over channels in random order; for a channel currently in group $\{A_j(x)\}_{j=1}^d$8, evaluate the objective change $\{A_j(x)\}_{j=1}^d$9 for every target group $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$0, reassigning if any move decreases the objective and updating $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$1. Fourth, repeat sweeps until no reassignment improves the objective or a maximum number of sweeps is reached, with the paper giving “e.g. 5 000” as a limit (Zhao et al., 1 Oct 2025).

Once the grouping is fixed, the method renders phrase-specific heat maps

$a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$2

and labels each map with phrase $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$3 (Zhao et al., 1 Oct 2025). This converts a monolithic saliency map into a set of semantically partitioned saliency components. In the paper’s examples, this enables explanations such as separate maps for “long-necked,” “gangly,” and “slender beak” in an ImageNet “American egret” prediction, or separate part-aligned phrases such as “pointed beak,” “yellow belly,” and “spotted wings” in CUB-200 (Zhao et al., 1 Oct 2025).

5. Empirical evaluations and reported results

The paper reports experiments on ImageNet, CLEVR, CUB, and DomainNet, and characterizes the resulting explanations as faithful and interpretable while preserving model fidelity (Zhao et al., 1 Oct 2025). The evaluation spans qualitative analysis, human interpretability assessment, controlled diagnostic tests, fine-grained attribute agreement, and a debiasing use case.

Setting	Reported result	Context
ImageNet-1K	User understanding improved + 37%	Human study on interpretability
CLEVR Diagnostic	Top-1 textual accuracy $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$4	Both frozen heads over 100 test images
CUB-200	86% of generated attributes matched ground-truth part-attribute annotations	Fine-grained textual agreement
CUB-200	Classification accuracy unchanged ($a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$5)	After adding TextCAM hooks
CLEVR shape debiasing	75.67% $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$6 84.33% (+8.66 pp)	After ablating color-dominant channels

On ImageNet-1K, the reported models are ResNet-50 and Swin, and the CAM families examined include Grad-CAM, Layer-CAM, Score-CAM, Eigen-CAM, and Finer-CAM (Zhao et al., 1 Oct 2025). The qualitative claim is that TextCAM consistently yields plausible phrases faithful to the underlying CAM regions. The human study reports that TextCAM rationales improved user understanding by $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$7.

On the CLEVR diagnostic, the evaluation uses two frozen heads, $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$8 for shape and $a_j(x)=\tfrac1{H\,W}\sum_{u,v} A_j(x)_{u,v},$9 for color, with held-out distractors described as yellow cylinders (Zhao et al., 1 Oct 2025). For a red ball, the shape model produces phrases such as “ball” and “round,” whereas the color model produces phrases such as “red” and “warm-colored,” despite the same spatial focus. The reported Top-1 textual accuracy is $M$0 for both heads over 100 test images. The paper also states that the spatial maps are insensitive to distractors, so the textual attribution is what reflects the correct factorized property.

On CUB-200, the paper reports that $M$1 of generated attributes matched ground-truth part-attribute annotations and that classification accuracy was unchanged within $M$2 after adding TextCAM hooks (Zhao et al., 1 Oct 2025). This is presented as evidence that the explanation mechanism does not materially perturb the base classifier while still recovering fine-grained descriptive content.

On DomainNet, ResNet-50 is trained on four domains—real, clipart, sketch, and quickdraw—and the ratio of color/shape/texture phrases in Top-1 TextCAM results is computed (Zhao et al., 1 Oct 2025). The reported trend is high shape reliance in sketch and quickdraw and high color reliance in real and clipart, matching domain priors.

6. Interpretive uses, diagnostics, and scope

A central claim of TextCAM is that textualized saliency supports interpretability diagnostics that are difficult to perform with spatial maps alone (Zhao et al., 1 Oct 2025). The paper explicitly identifies detection of spurious correlations as one such use case, since a conventional heat map may reveal that the model attends to a region without revealing whether the operative cue is the object itself, background color, or another attribute. By attaching attributes to CAM responses, TextCAM provides a mechanism for identifying which factor is being used within the highlighted region.

The reported debiasing experiment on a CLEVR shape classifier illustrates this use. Under color-biased data, original shape accuracy is reported as $M$3. After ablating color-dominant channels mined through TextCAM analysis, accuracy increases to $M$4, a gain of $M$5 percentage points, and the intervention is described as requiring no further training (Zhao et al., 1 Oct 2025). The paper interprets this as confirmation that TextCAM can guide targeted feature-space interventions.

The method is also stated to preserve model fidelity. On CUB-200, classification accuracy remains unchanged within $M$6 after adding TextCAM hooks (Zhao et al., 1 Oct 2025). In this sense, TextCAM is presented as an explanatory overlay rather than a retrained surrogate. The paper further summarizes the framework as augmenting any CAM variant by lifting channel activations into a CLIP-LDA semantic space, aggregating them via CAM weights, and retrieving sparse, diverse textual concepts, optionally grouped for fine-grained heat maps, while requiring no modification to the original vision model (Zhao et al., 1 Oct 2025).

A plausible implication is that TextCAM occupies an intermediate position between post hoc saliency analysis and concept-based explanation. It does not replace the underlying visual evidence map, but it does convert that evidence into a vocabulary-aligned representation that can be audited, compared across domains, and used for channel-level interventions. Within the scope reported by the paper, its primary contribution is therefore not new localization but semantic resolution of existing CAM explanations.