CLIP Surgery: Reliable CAM in CLIP

• CLIP Surgery is a training-free, inference-time modification of CLIP that employs dual-path architecture and value-to-value attention to produce more reliable CAMs.
• It replaces standard query-key attention and removes redundant cross-category features, addressing issues like background-focused attention and noisy activations.
• Empirical results demonstrate significant improvements in mIoU and mSC benchmarks on segmentation tasks without the need for additional fine-tuning.

CLIP Surgery is a training-free, inference-time modification to the CLIP architecture and features for reliable class activation mapping (CAM), introduced in "A Closer Look at the Explainability of Contrastive Language-Image Pre-training" (Li et al., 2023). It was proposed to address a specific failure mode of raw CLIP localization: direct patch-text similarity maps often highlight background rather than foreground, link semantically inconsistent regions, and produce spot-like noisy activations. The method combines an architectural intervention—value-to-value self-attention in a separate explanatory path, with feed-forward networks omitted from that path—with feature debiasing that subtracts redundant cross-category components from patch-text interactions. In later work, the phrase "CLIP Surgery" has also been reused more broadly for selective intervention on CLIP representations or optimization, notably in few-shot OOD detection and in CLIP fine-tuning for AI-generated image detection (Sun et al., 2024, Yan et al., 17 Nov 2025).

1. Definition, provenance, and scope

The primary referent of the term is the 2023 explainability method of Li et al., which treats CLIP not as a model to be retrained for localization, but as a model whose inference mechanics can be modified to yield more reliable patch-level evidence without further fine-tuning (Li et al., 2023). The paper’s central premise is that CLIP’s localization failures are not merely artifacts of a poor explainer. Rather, they are tied to how token relations and feature redundancies behave under CLIP’s image-text contrastive pretraining.

This scope is narrower than the broader universe of CLIP-based adaptation methods. Separate CLIP-based systems for surgical phase recognition, surgical scene segmentation, and fine-grained surgical action recognition—such as CauCLIP, CLIP-RL, and fine-CLIP—address sim-to-real transfer, dense segmentation, and zero-shot triplet recognition in the operative domain, but they are distinct from the method named CLIP Surgery (He et al., 6 Feb 2026, Ahmed et al., 6 Jul 2025, Sharma et al., 25 Mar 2025). A plausible implication is that the phrase is best understood as a named explainability and representation-intervention technique, rather than as a generic label for all CLIP methods used in surgery or medicine.

2. Diagnostic analysis of vanilla CLIP localization

The starting point is the raw patch-text similarity map obtained by comparing normalized image token features and text features, reshaping the resulting scores into image space, and resizing them to the input resolution:

$$\boldsymbol{M} = norm\Big( resize\Big( reshape\Big( \frac{\boldsymbol{F}_i}{\|\boldsymbol{F}_i\|_2} \cdot \Big(\frac{\boldsymbol{F}_t}{\|\boldsymbol{F}_t\|_2}\Big)^\top \Big) \Big) \Big).$$

Here, $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$ are image token features, $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$ are text features, $N_i$ is the number of image patches, $N_t$ is the number of text prompts or classes, and $C$ is the feature dimension (Li et al., 2023). This map is the most direct CLIP localization signal, since it avoids backpropagation-based explainers and uses CLIP’s own patch-text similarity.

The paper identifies three recurrent failure modes. The first is background-focused attention, or "opposite visualization," in which CLIP scores background regions higher than the object corresponding to the queried text. The second is inconsistent semantic regions, where token interactions connect semantically mismatched locations. The third is noisy activations on irrelevant positions, visible as scattered hotspots that persist across categories and even under an empty-string prompt. The paper argues that the first two are linked to CLIP’s original self-attention, while the third reflects redundant common components shared across categories.

The attention diagnosis is expressed through standard transformer attention,

$$Attn_{qk} = softmax(\boldsymbol{Q}\boldsymbol{K}^\top \cdot scale)\,\boldsymbol{V},$$

which the paper claims produces incorrect token relations for localization. The feature diagnosis is that the patch-text interaction includes category-agnostic or weakly category-correlated components that contaminate spatial evidence. These two diagnoses motivate the two-part intervention: architecture surgery and feature surgery.

3. Architecture surgery

Architecture surgery replaces query-key affinity with value-value affinity in the explanatory path:

$$Attn_{vv} = softmax(\boldsymbol{V}\boldsymbol{V}^\top \cdot scale)\,\boldsymbol{V}.$$

The paper’s claim is that this yields token relations based directly on the propagated value features, so that a token is linked more strongly to semantically consistent regions rather than to regions induced by learned query and key projections (Li et al., 2023). The intervention is not applied by simply overwriting the original CLIP stack. Instead, the method uses dual paths: an untouched original path and a new surgical path.

In the dual-path design, the original CLIP path is preserved for stable global recognition behavior, while a new path begins at depth $d$, applies $v$-$\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$0 self-attention in subsequent layers, and skips feed-forward networks in that path. The original path continues to produce the class-token feature used for global class weighting, whereas the new path produces patch tokens used for localization. The paper reports $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$1 as the default setting and states that the hyperparameters are not very sensitive. This architecture is explicitly training-free: the original weights are reused at inference, and no task-specific fine-tuning is required.

The omission of feed-forward networks is part of the method rather than an incidental simplification. The paper reports that self-attention outputs are closer to the final discriminative feature than feed-forward outputs, and that some feed-forward outputs behave like negative-label directions for explainability. This is why CLIP Surgery’s new path is attention-only. The result is a modified inference route intended to preserve attention-derived spatial structure while avoiding transformations that the paper argues are harmful for localization.

4. Feature surgery and the construction of reliable CAMs

Feature surgery addresses noisy activation by removing redundant common components from patch-text interactions. The method first forms a channelwise interaction tensor between normalized image and text features:

$\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$2

where $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$3 repeats image token features across the class dimension and $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$4 repeats text features across the patch dimension (Li et al., 2023). The paper then derives class weights from the original-path class token:

$\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$5

with $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$6 in the default setting. These weights emphasize classes that CLIP already considers likely.

The redundant component is estimated by a weighted mean across classes,

$\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$7

and subtracted before summing over channels:

$\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$8

The denoised similarity matrix $\boldsymbol{F}_i \in \mathbb{R}^{N_i \times C}$9 is then reshaped and resized exactly as in the raw map construction, yielding the final CAM (Li et al., 2023). In effect, the method interprets patch-text cosine similarity as a channelwise interaction, estimates the common cross-category component, and removes it.

This reliable CAM becomes the basis for several downstream procedures. For open-vocabulary semantic segmentation, the method increases image resolution, computes class maps, and applies an argmax over the class dimension. For open-vocabulary multi-label recognition, it substitutes the image patch features with the class-token feature $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$0, producing image-level class scores. For interactive segmentation with SAM, it converts the heatmap into point prompts by taking foreground points with score above $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$1 and background points from the same number of lowest-scoring locations. For a single sentence prompt rather than a class set, the paper replaces the redundant-feature estimate with a redundancy feature derived from an empty string. The paper also notes that feature surgery is mainly useful when score calibration across images or classes matters, and is less meaningful for single-label classification or rank-only settings.

5. Empirical behavior, benchmarks, and ablations

On explainability benchmarks, the main empirical effect is that CLIP Surgery replaces negative foreground-background score contrast with strongly positive contrast and substantially improves mIoU (Li et al., 2023).

Benchmark	Vanilla CLIP	CLIP Surgery
ImageNet-S50	27.87 mIoU / -18.84 mSC	62.41 mIoU / 36.50 mSC
VOC 2012	17.36 mIoU / -19.86 mSC	55.78 mIoU / 41.64 mSC
PASCAL Context	15.76 mIoU / -16.73 mSC	46.28 mIoU / 34.32 mSC
COCO 2017	9.74 mIoU / -23.37 mSC	35.23 mIoU / 35.43 mSC

The metric $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$2 is designed to detect opposite visualization; negative values indicate that background scores exceed foreground scores. The reported numbers therefore support the paper’s claim that raw CLIP often prefers background, whereas CLIP Surgery produces foreground-dominant maps. On VOC 2012 with ViT-B/16, for example, the paper reports 55.78 mIoU and 41.64 mSC, exceeding ECLIP’s 48.46 mIoU and 28.83 mSC while requiring no fine-tuning.

The downstream effects are also quantitative. For open-vocabulary semantic segmentation with ViT-B/16 and no training, the paper reports 29.3 mIoU on PASCAL Context, 21.9 on COCO Stuff, and 31.4 on Cityscapes. For open-vocabulary multi-label recognition on NUS-Wide, CLIP ViT-B/16 improves from 42.85 mAP under the softmax baseline to 47.19 mAP with surgery, and the abstract highlights a 4.41% mAP gain without training. The method is also reported to improve text-derived point prompting for SAM, especially at higher input resolution.

The ablations separate the two components. On PASCAL Context with ViT-B/16, original CLIP yields mSC $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$3, mIoU 15.76, and mAP 47.09; architecture surgery alone yields 31.15, 43.47, and 47.09; architecture plus feature surgery yields 34.32, 46.28, and 52.61. This supports the paper’s interpretation that architecture surgery primarily fixes opposite visualization, while feature surgery further suppresses noise and improves recognition. A second ablation shows that replacing only the last layer gives mSC 29.13, replacing multiple layers without dual paths gives $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$4, adding dual paths gives 32.30, and adding feed-forward networks back into the new path degrades performance to $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$5. The best attention-only dual-path configuration reaches 34.32 mSC.

6. Later reinterpretations, extensions, and terminological diffusion

A first line of reuse appears in CLIP-OS, where CLIP Surgery is repurposed as a feature-selection mechanism for few-shot OOD detection rather than as a standalone explainability method (Sun et al., 2024). CLIP-OS adopts the value-value attention formulation,

$\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$6

and uses it to build a surgery-based patch similarity map $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$7. It then compares this against the original CLIP map $\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$8 and defines a discrepancy-corrected masking rule,

$\boldsymbol{F}_t \in \mathbb{R}^{N_t \times C}$9

to separate the ID-relevant region $N_i$0 from the ID-irrelevant region $N_i$1. CLIP-OS further adds patch uniform convolution with a fixed $N_i$2 kernel of $N_i$3 entries and $N_i$4. In its ablation on CIFAR-100 average AUROC, the full method reaches 78.24, removing masking yields 76.93, and a Top-K rank baseline yields 76.59. Here, CLIP Surgery functions as a foreground-disentangling operator upstream of synthetic OOD generation and unknown-aware prompt learning.

A second line of reinterpretation appears in DGS-Net, which uses the phrase "CLIP surgery" in a broader optimization-time sense during LoRA-based CLIP fine-tuning for AI-generated image detection (Yan et al., 17 Nov 2025). DGS-Net defines the harmful direction as the positive half-space of the text gradient and the beneficial direction as the negative half-space of the frozen teacher image gradient. The image-feature gradient is then explicitly replaced by

$N_i$5

This is described as editing the gradient before it reaches the trainable CLIP image encoder. Unlike the original CLIP Surgery, which is training-free and inference-time, DGS-Net is a training-time intervention designed to preserve transferable priors while suppressing task-irrelevant components. The paper reports experiments on 50 generative models and states that the method outperforms state-of-the-art approaches by an average margin of 6.6, with no extra test-time cost from the teacher or text branch.

This suggests a broader historical pattern. In its original sense, CLIP Surgery denotes a specific training-free method for reliable CAM and open-vocabulary dense prediction. In later work, the phrase has also become a more general descriptor for selective intervention on CLIP—sometimes at the level of patch similarities, sometimes at the level of gradient flow. The continuity across these usages is the idea of not accepting CLIP’s pretrained representations or updates wholesale, but instead identifying and removing components regarded as harmful while retaining or injecting components regarded as useful.