PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation

Abstract: Training-free open-vocabulary semantic segmentation (OVSS) promises rapid adaptation to new label sets without retraining. Yet, many methods rely on heavy post-processing or handle text and vision in isolation, leaving cross-modal geometry underutilized. Others introduce auxiliary vision backbones or multi-model pipelines, which increase complexity and latency while compromising design simplicity. We present PEARL, \textbf{\underline{P}}rocrust\textbf{\underline{e}}s \textbf{\underline{a}}lignment with text-awa\textbf{\underline{r}}e \textbf{\underline{L}}aplacian propagation, a compact two-step inference that follows an align-then-propagate principle. The Procrustes alignment step performs an orthogonal projection inside the last self-attention block, rotating keys toward the query subspace via a stable polar iteration. The text-aware Laplacian propagation then refines per-pixel logits on a small grid through a confidence-weighted, text-guided graph solve: text provides both a data-trust signal and neighbor gating, while image gradients preserve boundaries. In this work, our method is fully training-free, plug-and-play, and uses only fixed constants, adding minimal latency with a small per-head projection and a few conjugate-gradient steps. Our approach, PEARL, sets a new state-of-the-art in training-free OVSS without extra data or auxiliary backbones across standard benchmarks, achieving superior performance under both with-background and without-background protocols.

• The paper introduces a training-free method that leverages Procrustes alignment within CLIP's attention mechanism to stabilize patch-text matching.
• It employs text-aware Laplacian propagation to refine segmentation masks by integrating image gradients with text-guided semantic grouping.
• Experimental results show that PEARL outperforms traditional methods, especially in segmenting small and thin objects, while reducing computational overhead.

Geometry Alignment in Semantic Segmentation

Introduction to Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation (OVSS) aims to segment images using label sets specified at inference time, bridging the semantic gap between visual data and natural language descriptions. Traditional semantic segmentation methods typically restrict models to a closed set of categories, requiring retraining and reannotation to incorporate new categories. OVSS promises flexibility without retraining by utilizing vision-LLMs, such as CLIP and ALIGN, to create shared image-text feature spaces. Despite these advances, existing methods often involve complex pipelines with additional backbones and extensive post-processing, which might compromise inference efficiency.

The paper introduces a novel approach, PEARL, focused on training-free semantic segmentation. It adopts an align-then-propagate methodology that combines Procrustes alignment and text-aware Laplacian propagation. PEARL modifies the inference process without retraining, using systematic alignment within attention scores and refinement via graph-based propagation.

Figure 1: Attention visualization at selected query points showcasing the compact and consistent object attention achieved by PEARL compared to other methods.

Methodology and Implementation

Procrustes Alignment

The methodology begins with systematic alignment known as Procrustes alignment, which is implemented within the self-attention mechanisms of the CLIP architecture. This alignment involves orthogonal projections that ensure keys are rotated toward the query subspace, effectively stabilizing the token geometry and improving patch/text matching reliability. This alignment is performed in two steps, weighted centering to mitigate high-norm background tokens' influence, followed by a rotated orthogonal transformation preserving local magnitudes and subsequently recalibrating the feature space used for patch-text cosine similarity.

Text-aware Laplacian Propagation

In the final stage, PEARL refines segmentation masks using text-aware Laplacian propagation. This step leverages a confidence-weighted and text-guided graph-based method to strategize semantic propagation. Image gradients are used to safeguard boundaries during propagation. A small grid keeps computational complexity low, refining logits with weighted data terms incorporating pixel confidence and text agreement priors. Significantly, this process uses text relations to direct context-sensitive grouping, enforcing semantic relationships during mask refinement.

Figure 2: PEARL framework illustrating the align-then-propagate process.

Experimental Results

Comparison with State-of-the-Art

PEARL achieves state-of-the-art performance in training-free OVSS benchmarks, consistently outperforming traditional approaches that rely on heavy auxiliary backbones or additional training. Its architecture allows heightened focus on small and thin object structures and maintains performance over extensive "stuff" regions.

Limitations and Future Directions

PEARL's reliance on text prompts makes it sensitive to prompt quality and label nomenclature. It does not accommodate instance-aware segmentations effectively, a common limitation in training-free methods. Improvements could include adaptive grids and enhanced instance discrimination capabilities, propelling PEARL further in complex real-world scenes.

Conclusion

PEARL stands out in the domain of training-free OVSS by simplifying the inference process, reducing latency without sacrificing accuracy. This advancement presents a valuable contribution to semantic segmentation tasks, emphasizing the significance of geometry alignment in enhancing model capabilities within text-aware frameworks.