FreeCP: Training-Free Purification for OVSS

• FreeCP is a training-free framework for open-vocabulary semantic segmentation that purifies the candidate class set to improve dense predictions.
• It employs a two-stage purification with redundancy and ambiguity removal to resolve false activations and overlapping semantic cues.
• Empirical results demonstrate consistent mIoU gains across eight benchmarks, notably boosting MaskCLIP performance by +10.9 mIoU.

FreeCP is a training-free class purification framework for open-vocabulary semantic segmentation (OVSS) that operates on top of CLIP-based dense prediction pipelines. Its stated purpose is to improve segmentation under inference-time overcomplete vocabularies by addressing two specific error modes—class redundancy and visual-language ambiguity—before final decoding. Rather than training a new segmentor, FreeCP purifies the active class set and the corresponding activation maps, then applies argmax over the purified activations to obtain segmentation predictions. It is presented as a plug-and-play module for existing training-free OVSS methods and is evaluated across eight benchmarks using mIoU as the principal metric (Chen et al., 1 Aug 2025).

1. Problem formulation in training-free OVSS

Open-vocabulary semantic segmentation seeks to assign pixels to arbitrary text labels, typically by combining vision-language representations with dense activation generation. In the formulation associated with FreeCP, the central difficulty is not merely producing class activation maps, but doing so when inference is performed over a long candidate vocabulary even though only a small subset of classes is actually present in a given image (Chen et al., 1 Aug 2025).

The paper isolates two failure modes. Class redundancy arises when classes absent from the image nonetheless receive activation, creating false positives and distorting the final argmax. Visual-language ambiguity arises when semantically similar categories activate the same or similar regions, such as leaves, bush, and tree, or river and water, causing uncertain local assignments. The key thesis is that refinement alone is insufficient in this setting, because class-agnostic propagation can amplify spurious responses if irrelevant classes remain in the candidate set. This motivates a purification stage prior to final segmentation.

Within this framing, FreeCP is not a replacement for dense OVSS inference. It is a pre-decoding control mechanism that filters and resolves class hypotheses on a per-image basis. A plausible implication is that the method shifts the main source of error from feature extraction to class-set management, especially in settings with large vocabularies and semantically crowded label spaces.

2. CLIP-based activation generation and refinement

FreeCP is built on a ViT-based CLIP encoder. The text encoder output is

$\mathbf{T}=\mathcal{E}_{T}(\mathcal{Z}) \in \mathbbm{R}^{K\times d},$

where $K$ is the number of class prompts and $d$ is the feature dimension. The image encoder produces patch tokens $\mathbf{F}^{p} \in \mathbbm{R}^{N\times d}$ and a class token $\mathbf{F}^{c} \in \mathbbm{R}^{1\times d}$. Initial dense activations are obtained by comparing patch tokens with text embeddings via cosine similarity and normalizing across classes:

$$\mathbf{M}_{j} = Reshape\!\left(\frac{\exp(Sim(\mathbf{F}^{p}, \mathbf{T}_{j}))}{\sum_{j}{\exp(Sim(\mathbf{F}^{p}, \mathbf{T}_{j}))}}\right).$$

This yields the initial class-wise activation maps $\mathbf{M}$ (Chen et al., 1 Aug 2025).

To improve localization, the method derives an image self-affinity matrix from CLIP self-attention:

$SA = \frac{1}{L} \sum_{l}^{L} \psi(A_{l}),$

where $A_l$ is the attention matrix from layer $l$, $K$0 is the number of layers used, and $K$1 resizes attention maps to a common spatial resolution. Refinement is then performed by propagating each class activation map through this affinity structure:

$K$2

The intended effect is completion of sparse or incomplete object regions through semantically related patches. However, the paper emphasizes that this same mechanism can degrade performance when applied over the full vocabulary, because irrelevant classes are also reinforced. This is one of the central empirical premises behind FreeCP: refinement is useful only when conditioned on a purified class set.

3. Spatial consistency and the two-stage purification mechanism

The core signal used throughout FreeCP is Spatial Consistency (SC), defined as IoU between activation maps:

$K$3

Here, $K$4 and $K$5 are activation maps and $K$6 denotes element-wise multiplication. FreeCP uses this single measure in two distinct ways: first to remove redundant classes, and then to identify ambiguous ones (Chen et al., 1 Aug 2025).

In Redundancy Purification (RP), each class is scored by comparing its original and refined activations:

$K$7

If $K$8, the class is removed. The interpretation given is that a class genuinely present in the image should remain spatially stable under refinement, whereas a redundant class is more likely to drift or absorb spurious regions. The output of RP is a reduced class set $K$9.

In Ambiguity Purification (AP), FreeCP computes inter-class spatial consistency among the remaining refined maps:

$d$0

Pairs above the threshold $d$1 are marked as ambiguous:

$d$2

Connected components are then found using DFS to form ambiguity groups. This construction treats ambiguity as a graph over classes whose refined activations overlap strongly. The procedure is notable in that it does not define ambiguity semantically in the text space alone; it defines it through spatial competition in the image.

4. Local ambiguity elimination and final decoding

Once ambiguity groups have been formed, FreeCP resolves them locally rather than globally. For each ambiguity group, the method averages the ambiguous class maps to localize the uncertain region, extracts a bounding box around the high-response area, crops the original image, and resizes the crop, for example to $d$3. The crop is then passed through CLIP’s image encoder to obtain a local visual feature $d$4 (Chen et al., 1 Aug 2025).

The competing classes are represented not only by their class names but by fine-grained textual descriptions generated by an LLM in advance. The paper uses Vicuna-13b-1.5, with 3 prompts and 5 answers per prompt, yielding 15 descriptions per class; the final text feature is the average of those 15 descriptions. Local disambiguation is then performed through similarity maximization:

$d$5

An important design detail is that AP does not globally discard classes. Classes that lose the local comparison are set to zero only within the ambiguous local region. This preserves the possibility that the same class may be valid elsewhere in the image.

After RP and AP, FreeCP retains the purified activation maps $d$6 and obtains the final segmentation by applying argmax over the purified class activations. This suggests a decomposition of OVSS inference into two stages: dense evidence generation followed by class-space purification. The method’s contribution lies primarily in the second stage.

5. Empirical evaluation and reported performance

FreeCP is evaluated on eight benchmarks: PASCAL VOC 21 (VOC21), PASCAL VOC 20 (VOC20), PASCAL-Context 60 (PC60), PASCAL-Context 59 (PC59), MS COCO Object, MS COCO Stuff, ADE20K, and Cityscapes. The reported metric is mIoU throughout. The default backbone is ViT-B/16 CLIP, with additional experiments on ViT-L/14, OpenCLIP ViT-L/14, and ResNet-50x16. Inference resizes the image shorter side to 448 px, or 560 px for Cityscapes, and uses a sliding window of size 384 with stride 112. The reported results do not use PAMR or denseCRF. Hardware is listed as 8 × NVIDIA RTX 3090 GPUs (Chen et al., 1 Aug 2025).

The paper compares FreeCP-enhanced variants of several training-free OVSS baselines, including MaskCLIP, GEM, ClearCLIP, and SCLIP. The reported average gains are +10.9 mIoU for MaskCLIP + FreeCP, +3.7 mIoU for GEM + FreeCP, +4.1 mIoU for ClearCLIP + FreeCP, and +3.9 mIoU for SCLIP + FreeCP. Example average results from Table 2 are:

Method	Baseline average mIoU	With FreeCP
SCLIP	38.2	42.1
MaskCLIP	30.3	41.2
ClearCLIP	38.1	42.2
GEM	38.3	42.0

The improvements are reported as consistent across all eight benchmarks. The paper further states that FreeCP can achieve performance comparable to methods that use extra models such as DINOv2, SAM, or diffusion models, while remaining training-free and avoiding extra heavy auxiliary models. Within the reported results, especially strong gains are observed for MaskCLIP and for complex datasets such as ADE20K and COCO Stuff, where redundancy and ambiguity are more severe.

6. Ablations, interpretation, and limitations

The ablation study centers on the order and composition of purification stages. On VOC21, the paper reports: baseline 59.8, +Refine 27.5, RP 65.8, AP 37.7, AP-RP 57.3, and RP-AP 65.8. The stated conclusion is that refinement alone hurts badly, RP provides the largest gain, AP adds additional gains after RP, and RP-AP is better than AP-RP (Chen et al., 1 Aug 2025).

This result addresses a likely misconception about affinity propagation in OVSS: more refinement is not necessarily better. In the FreeCP analysis, refinement without vocabulary control is detrimental because it amplifies false positives under full-vocabulary inference. The method therefore treats redundancy removal as a prerequisite for meaningful ambiguity resolution.

Additional ablations test textual description sources—Template, Vicuna, and GPT-3.5—and report that fine-grained descriptions consistently help, with Vicuna tending to work best. Affinity-source comparisons among MaskFormer, DINO, SAM, and CLIP show that CLIP self-attention performs best. The paper interprets this as better alignment between CLIP attention and the text-image semantics relevant for OVSS. FreeCP is also reported to improve performance across ViT-L/14, OpenCLIP ViT-L/14, and ResNet-50x16; for ResNet, where attention is unavailable, DINO features are used as the refinement affinity representation.

The limitations stated or implied are operational rather than conceptual. Thresholds $d$7 and $d$8 are set empirically according to dataset complexity, with lower thresholds for ADE/Stuff, balanced settings for Cityscapes/Context, and higher thresholds for VOC/Object. AP introduces extra inference overhead through local crop extraction and fine-grained text comparison. The method depends on the quality of CLIP activations and on the utility of the precomputed fine-grained descriptions. The framework is also heuristic rather than end-to-end learned. Even so, the reported results suggest that, in training-free OVSS, adaptive purification of the candidate vocabulary can be at least as consequential as changes to the dense prediction backbone or refinement operator itself.