CLIPSym: Dense Symmetry Detection Framework

• CLIPSym is a symmetry-detection framework that adapts a pretrained vision-language model to generate dense heatmaps for reflection and rotation in natural images.
• It integrates CLIP’s image and text encoders with a rotation-equivariant decoder and Semantic-Aware Prompt Grouping to inject symmetry-relevant semantic cues.
• Empirical evaluations on standard benchmarks demonstrate that CLIPSym outperforms previous methods by effectively combining multimodal pretraining with geometric inductive bias.

Searching arXiv for CLIPSym and closely related symmetry-detection work. CLIPSym is a symmetry-detection framework that adapts a pretrained CLIP model to dense prediction of reflection axes and rotation centers in natural images. It combines CLIP’s image and language encoders with a rotation-equivariant decoder based on a hybrid of Transformer and $G$-Convolution, and introduces a prompting method called Semantic-Aware Prompt Grouping (SAPG) to inject symmetry-relevant semantic cues through grouped frequent-object prompts. The method is evaluated on the standard symmetry-detection datasets DENDI, SDRW, and LDRS, where it is reported to outperform prior state of the art, while ablations isolate the effects of CLIP pre-training, language conditioning, and the equivariant decoder (Yang et al., 19 Aug 2025).

1. Definition and task formulation

CLIPSym addresses dense symmetry detection in natural images. For an input image $I \in \mathbb{R}^{H\times W\times 3}$, it predicts a symmetry heatmap

$\hat{S}_I \in [0,1]^{H\times W},$

with each pixel indicating the likelihood of belonging to a symmetry structure (Yang et al., 19 Aug 2025).

The framework treats two related but distinct tasks. In reflection symmetry detection, the heatmap marks pixels lying on reflection axes. In rotation symmetry detection, it marks pixels corresponding to rotation centers. The output is therefore a dense per-pixel probability map rather than an explicit parametric representation such as a line equation, center coordinate list, or rotation order. This design places CLIPSym in the family of dense prediction systems rather than structured geometric parsers (Yang et al., 19 Aug 2025).

The paper motivates the task as difficult because natural images contain cluttered backgrounds, partial occlusions, viewpoint and pose changes, imperfect or approximate symmetry, and limited annotated training data. A central premise is that CLIP’s large-scale vision-language pre-training may encode semantic regularities correlated with symmetric structure, and that such regularities can be exploited even though symmetry itself is geometric (Yang et al., 19 Aug 2025).

2. Architectural composition

CLIPSym consists of five principal stages: a CLIP image encoder, a CLIP text encoder, FiLM-based multimodal conditioning, a Transformer with prompt aggregation, and a rotation-equivariant upsampling decoder (Yang et al., 19 Aug 2025).

The visual backbone is pretrained CLIP ViT-B/16. Images are resized or padded to $417\times 417$, preserving aspect ratio and zero-padding the shorter side. With patch size $16\times16$, the encoder produces a $26\times26$ patch grid, denoted

$z_I = \{z_{p_{ij}}\},$

where $z_{p_{ij}} \in \mathbb{R}^d$ is the feature at patch location $(i,j)\in [M]^2$ and $M=26$ (Yang et al., 19 Aug 2025).

The text branch encodes a prompt set

$I \in \mathbb{R}^{H\times W\times 3}$0

into

$I \in \mathbb{R}^{H\times W\times 3}$1

These text embeddings do not serve as classification prototypes; instead, they modulate image tokens through FiLM. For each prompt $I \in \mathbb{R}^{H\times W\times 3}$2, CLIPSym computes

$I \in \mathbb{R}^{H\times W\times 3}$3

with

$I \in \mathbb{R}^{H\times W\times 3}$4

where $I \in \mathbb{R}^{H\times W\times 3}$5 and $I \in \mathbb{R}^{H\times W\times 3}$6 are linear layers, and the FiLM dimension is set to $I \in \mathbb{R}^{H\times W\times 3}$7 (Yang et al., 19 Aug 2025).

Each text-conditioned token set is then processed by a Transformer: $I \in \mathbb{R}^{H\times W\times 3}$8 The Transformer uses three self-attention layers. Prompt-conditioned outputs are aggregated by a convex combination

$I \in \mathbb{R}^{H\times W\times 3}$9

This yields a single multimodal patch representation $\hat{S}_I \in [0,1]^{H\times W},$0 for decoding (Yang et al., 19 Aug 2025).

A plausible implication is that CLIPSym treats language as a global semantic prior over the image rather than as localized textual supervision. The architecture does not compute token-patch cross-attention in the style of dense grounding models; instead, it uses prompt-conditioned affine modulation followed by global self-attention over visual tokens.

3. Equivariant decoding and geometric structure

The decoder is the explicitly geometric component of CLIPSym. After prompt aggregation, tokens are rearranged into a spatial feature map

$\hat{S}_I \in [0,1]^{H\times W},$1

This map is then lifted to the roto-translation group

$\hat{S}_I \in [0,1]^{H\times W},$2

by concatenating rotated copies: $\hat{S}_I \in [0,1]^{H\times W},$3 More explicitly,

$\hat{S}_I \in [0,1]^{H\times W},$4

or compactly $\hat{S}_I \in [0,1]^{H\times W},$5 (Yang et al., 19 Aug 2025).

On this lifted representation, CLIPSym applies steerable $\hat{S}_I \in [0,1]^{H\times W},$6-convolutions: $\hat{S}_I \in [0,1]^{H\times W},$7 The decoder uses three layers of $\hat{S}_I \in [0,1]^{H\times W},$8-convolution together with $\hat{S}_I \in [0,1]^{H\times W},$9 bilinear upsampling, and mean-pools over orientation channels to produce the final heatmap $417\times 417$0 (Yang et al., 19 Aug 2025).

The paper formalizes a rotation action on image tokens as

$417\times 417$1

where $417\times 417$2 is the permutation induced by rotating the patch grid. It then states the equivariance relation

$417\times 417$3

The proof relies on the permutation equivariance of FiLM and Transformer blocks under grid rotations, together with the equivariance of the $417\times 417$4-convolutional upsampler (Yang et al., 19 Aug 2025).

An important point is that this formal guarantee concerns rotation equivariance, not reflection equivariance. Reflection detection is handled by the same architecture, but the theorem in the paper addresses cyclic rotations.

4. Semantic-Aware Prompt Grouping

SAPG is the method’s main language-side contribution. Rather than using abstract prompts such as “reflection axis” or “symmetry axes in the image” alone, CLIPSym constructs prompts by concatenating frequent object names extracted from the dataset. A prompt takes the form

$417\times 417$5

For example, with $417\times 417$6, a prompt may be “apple cloud table” (Yang et al., 19 Aug 2025).

The object vocabulary is derived using Grounded-SAM, which extracts the 2081 frequent object classes from DENDI. The prompt set is fixed across images, but the CLIP text encoder is fine-tuned during training, so the text embeddings remain task-adaptive (Yang et al., 19 Aug 2025).

SAPG has two grouping levels. First, multiple object words are grouped within each prompt. Second, multiple prompts are used jointly, with learned aggregation weights $417\times 417$7. The paper reports a search over

$417\times 417$8

and finds the best setting to be

$417\times 417$9

for both reflection and rotation experiments (Yang et al., 19 Aug 2025).

The DENDI reflection ablation illustrates the effect of prompt design:

Prompting strategy	F1
“reflection axis”	64.4
“symmetry axes in the image”	64.8
single prompt of frequent object classes ($16\times16$0)	65.8
multi-prompt $16\times16$1	65.3
multi-prompt $16\times16$2	66.5
multi-prompt $16\times16$3	65.9

These results indicate that grouped frequent-object prompts outperform abstract symmetry phrases, and that increasing prompt count or prompt length beyond the $16\times16$4 regime does not continue to improve performance (Yang et al., 19 Aug 2025).

The paper’s interpretation is that frequent-object prompts provide stronger initialization in CLIP’s language space because they are closer to CLIP’s pre-training distribution. This suggests that SAPG does not encode symmetry explicitly in language; rather, it exploits object semantics statistically associated with symmetric structure.

5. Optimization, datasets, and evaluation protocol

Training uses a pixelwise $16\times16$5-focal loss: $16\times16$6 where

$16\times16$7

The balancing coefficient is $16\times16$8 for reflection and $16\times16$9 for rotation, with $26\times26$0 in both tasks (Yang et al., 19 Aug 2025).

Both the CLIP image encoder and the CLIP text encoder are fine-tuned. The optimizer is Adam, training runs for 500 epochs with batch size 16, and the initial learning rate is chosen from

$26\times26$1

with $26\times26$2 selected. Reflection detection uses an exponential-decay schedule with decay rate 0.1, whereas rotation detection uses a constant learning rate. Training augmentation includes random rotations by multiples of $26\times26$3, random rotations in $26\times26$4, and color jittering (Yang et al., 19 Aug 2025).

The evaluation uses three standard symmetry datasets. DENDI contains both reflection and rotation subsets; the reflection subset has 2493 images with splits 1750/374/369 for train/val/test, and the rotation subset has 2079 images with splits 1459/313/307. SDRW is used for reflection only, with 51 train and 70 test images. LDRS contains 1110/127/240 train/val/test images (Yang et al., 19 Aug 2025).

The principal metric is

$26\times26$5

computed by thresholding predicted heatmaps against binary ground-truth maps. The paper also reports a Robustness-score, defined as F1 under transformed inputs, and a Consistency-score

$26\times26$6

where lower is better (Yang et al., 19 Aug 2025).

6. Empirical performance, ablations, and significance

On DENDI, CLIPSym with the equivariant decoder and CLIP pre-training achieves

$26\times26$7

for reflection and

$26\times26$8

for rotation. The reproduced EquiSym baseline reports

$26\times26$9

while the originally reported EquiSym numbers are 64.5 / 22.5. The paper accordingly reports gains of +4.8 / +3.1 over the reproduced baseline and +2.0 / +2.6 over the originally reported EquiSym figures (Yang et al., 19 Aug 2025).

On reflection-only benchmarks, CLIPSym reports

$z_I = \{z_{p_{ij}}\},$0

on SDRW,

$z_I = \{z_{p_{ij}}\},$1

on LDRS, and

$z_I = \{z_{p_{ij}}\},$2

on the Mixed benchmark, exceeding EquiSym’s 48.2 / 37.7 / 41.1 in the same settings (Yang et al., 19 Aug 2025).

The ablations identify three main sources of improvement. First, CLIP pre-training is decisive: CLIPSym trained from scratch attains only 32.1 / 4.7 on DENDI reflection/rotation, whereas the CLIP-pretrained image-only variant reaches 63.7 / 17.7 (Yang et al., 19 Aug 2025). Second, language helps beyond image-only CLIP: adding SAPG and text conditioning raises DENDI performance from 63.7 / 17.7 to 66.5 / 25.1, with particularly large gains in rotation detection (Yang et al., 19 Aug 2025). Third, the equivariant decoder contributes beyond a non-equivariant alternative, improving DENDI from 62.9 / 24.2 to 66.5 / 25.1 and also improving reflection scores on SDRW, LDRS, and Mixed (Yang et al., 19 Aug 2025).

The trainability ablation shows that fine-tuning the image encoder matters most, but text-side adaptation still contributes. On DENDI reflection, the four configurations yield 59.4 with both encoders frozen, 58.9 with text trainable and image frozen, 65.3 with image trainable and text frozen, and 66.5 with both trainable (Yang et al., 19 Aug 2025).

Robustness analysis further separates CLIP feature quality from architectural equivariance. On DENDI reflection under random rotations, CLIPSym non-equivariant achieves robustness 58.3 and consistency 0.093, while CLIPSym equivariant reaches 59.7 and 0.082; both outperform EquiSym’s 57.1 and 0.244. This suggests that CLIP pre-training contributes practical rotation robustness even before the explicit equivariant decoder is applied (Yang et al., 19 Aug 2025).

A plausible implication is that CLIPSym’s gains arise from a specific division of labor: CLIP supplies semantically enriched and transformation-tolerant patch features, SAPG adds object-level priors that bias attention toward symmetric structures, and the $z_I = \{z_{p_{ij}}\},$3-convolutional decoder restores a geometric inductive bias that CLIP alone does not provide.

7. Interpretation, limitations, and position within symmetry detection

CLIPSym is notable because it frames symmetry detection as a problem at the intersection of geometry and semantics. The method does not claim that symmetry is linguistic; instead, it exploits the fact that large-scale caption corpora often encode shape- and object-related regularities. The paper reports that around 10% of captions in CLIP-scale corpora contain shape or symmetry cues such as “rectangle,” “circle,” and “oval,” and provides frequencies for related words including “ring,” “line,” “arc,” “ball,” and “square” (Yang et al., 19 Aug 2025). This suggests that language supervision can act as an indirect prior over likely symmetric entities.

The method’s limitations are equally clear. SAPG is heuristic: prompt construction is based on frequent-object extraction and hyperparameter search rather than a formal optimization principle. The decoder has a formal rotation-equivariance result, but no analogous theorem for reflections. The output remains a dense heatmap rather than a structured symbolic description of symmetry, so CLIPSym does not directly infer axis parameters, symmetry order, or decomposition into symmetric parts. Performance is also highly dependent on large-scale pre-training: the from-scratch ablation shows that the architecture alone is insufficient under current dataset sizes (Yang et al., 19 Aug 2025).

The paper additionally points to dataset quality as a substantial source of failure. The qualitative error analysis attributes some missed detections to annotation inconsistency, ambiguous rotation centers, and missing labels for objects that appear symmetric to human observers (Yang et al., 19 Aug 2025). This suggests that remaining error is not solely architectural.

Within the broader symmetry-detection literature, CLIPSym can be understood as a multimodal extension of dense equivariant detection: it preserves the dense heatmap formalism and equivariant decoding tradition while importing CLIP-scale semantic priors through prompt-conditioned feature modulation. Its core claim is therefore narrower than a general statement about language-guided geometry. It shows, more specifically, that pretrained vision-language representations can improve symmetry detection when paired with a decoder that respects geometric structure and a prompting scheme that avoids overly abstract symmetry text (Yang et al., 19 Aug 2025).