---
title: 'CLIPSym: Dense Symmetry Detection Framework'
url: https://www.emergentmind.com/topics/clipsym
type: topic
---

# CLIPSym: Dense Symmetry Detection Framework

Searching arXiv for CLIPSym and closely related symmetry-detection work.
CLIPSym is a symmetry-detection framework that adapts a pretrained CLIP model to dense prediction of reflection axes and rotation centers in natural images. It combines CLIP’s image and language encoders with a rotation-equivariant decoder based on a hybrid of Transformer and \(G\)-Convolution, and introduces a prompting method called Semantic-Aware Prompt Grouping (SAPG) to inject symmetry-relevant semantic cues through grouped frequent-object prompts. The method is evaluated on the standard symmetry-detection datasets DENDI, SDRW, and LDRS, where it is reported to outperform prior state of the art, while ablations isolate the effects of CLIP pre-training, language conditioning, and the equivariant decoder [2508.14197].

## 1. Definition and task formulation

CLIPSym addresses **dense symmetry detection in natural images**. For an input image \(I \in \mathbb{R}^{H\times W\times 3}\), it predicts a symmetry heatmap
\[
\hat{S}_I \in [0,1]^{H\times W},
\]
with each pixel indicating the likelihood of belonging to a symmetry structure [2508.14197].

The framework treats two related but distinct tasks. In **reflection symmetry detection**, the heatmap marks pixels lying on reflection axes. In **rotation symmetry detection**, it marks pixels corresponding to rotation centers. The output is therefore a dense per-pixel probability map rather than an explicit parametric representation such as a line equation, center coordinate list, or rotation order. This design places CLIPSym in the family of dense prediction systems rather than structured geometric parsers [2508.14197].

The paper motivates the task as difficult because natural images contain cluttered backgrounds, partial occlusions, viewpoint and pose changes, imperfect or approximate symmetry, and limited annotated training data. A central premise is that CLIP’s large-scale vision-language pre-training may encode semantic regularities correlated with symmetric structure, and that such regularities can be exploited even though symmetry itself is geometric [2508.14197].

## 2. Architectural composition

CLIPSym consists of five principal stages: a CLIP image encoder, a CLIP text encoder, FiLM-based multimodal conditioning, a Transformer with prompt aggregation, and a rotation-equivariant upsampling decoder [2508.14197].

The visual backbone is pretrained **CLIP ViT-B/16**. Images are resized or padded to \(417\times 417\), preserving aspect ratio and zero-padding the shorter side. With patch size \(16\times16\), the encoder produces a \(26\times26\) patch grid, denoted
\[
z_I = \{z_{p_{ij}}\},
\]
where \(z_{p_{ij}} \in \mathbb{R}^d\) is the feature at patch location \((i,j)\in [M]^2\) and \(M=26\) [2508.14197].

The text branch encodes a prompt set
\[
T=\{t_1,t_2,\dots\}
\]
into
\[
z_T=\{z_{t_1},z_{t_2},\dots\},\qquad z_{t_i}\in\mathbb{R}^d.
\]
These text embeddings do not serve as classification prototypes; instead, they modulate image tokens through FiLM. For each prompt \(t\), CLIPSym computes
\[
z_{I|t}=\mathrm{FiLM}(z_t,z_I)=\{z_{p_{ij}|t}\mid (i,j)\in[M]^2\},
\]
with
\[
z_{p_{ij}|t}=\gamma(z_t)\odot z_{p_{ij}}+\beta(z_t),
\]
where \(\gamma(\cdot)\) and \(\beta(\cdot)\) are linear layers, and the FiLM dimension is set to \(d=64\) [2508.14197].

Each text-conditioned token set is then processed by a Transformer:
\[
\hat{z}_{I|t}=D_{\mathrm{TF}}(z_{I|t})=\{\hat{z}_{p_{ij}|t}\mid (i,j)\in[M]^2\}.
\]
The Transformer uses three self-attention layers. Prompt-conditioned outputs are aggregated by a convex combination
\[
\bar{z}_{p_{ij}}=\sum_{t\in T}\alpha_t \hat{z}_{p_{ij}|t},
\qquad \alpha_t\ge 0,\quad \sum_t \alpha_t=1.
\]
This yields a single multimodal patch representation \(\bar z_I\) for decoding [2508.14197].

A plausible implication is that CLIPSym treats language as a global semantic prior over the image rather than as localized textual supervision. The architecture does not compute token-patch cross-attention in the style of dense grounding models; instead, it uses prompt-conditioned affine modulation followed by global self-attention over visual tokens.

## 3. Equivariant decoding and geometric structure

The decoder is the explicitly geometric component of CLIPSym. After prompt aggregation, tokens are rearranged into a spatial feature map
\[
f \triangleq \mathrm{Grid}(\bar{z}_I)\in \mathbb{R}^{d\times M\times M},
\qquad f[:,i,j]=\bar{z}_{p_{ij}}.
\]
This map is then lifted to the roto-translation group
\[
G=\mathbb{Z}_M^2\rtimes C_n
\]
by concatenating rotated copies:
\[
f^{\uparrow}\triangleq \mathrm{Concat}\left([r_\theta f;\ \forall \theta\in C_n]\right).
\]
More explicitly,
\[
f^\uparrow[i,\theta,x,y]=f[i,x',y']
\quad\text{where}\quad
\begin{pmatrix}x'\\y'\end{pmatrix}
=
\begin{pmatrix}
\cos\theta & \sin\theta\\
-\sin\theta & \cos\theta
\end{pmatrix}
\begin{pmatrix}x\\y\end{pmatrix},
\]
or compactly \([x',y']=r_{-\theta}(x,y)\) [2508.14197].

On this lifted representation, CLIPSym applies steerable \(G\)-convolutions:
\[
(f^\uparrow \star_G \psi)[\theta,x,y]
=
\sum_{\theta' \in G}\sum_{(x',y')\in [M]^2}
f^\uparrow[\theta',x',y']\,
\psi\!\left[\theta'-\theta,\,
r_{-\theta}(x'-x,y'-y)\right].
\]
The decoder uses three layers of \(G\)-convolution together with \(4\times\) bilinear upsampling, and mean-pools over orientation channels to produce the final heatmap \(\hat S_I\) [2508.14197].

The paper formalizes a rotation action on image tokens as
\[
r_\theta z_I \triangleq \{r_\theta z_{p_{ij}}\}=\{z_{p_{\pi_\theta(ij)}}\},
\]
where \(\pi_\theta\) is the permutation induced by rotating the patch grid. It then states the equivariance relation
\[
D(r_\theta z_I,z_T)=r_\theta \hat{S}_I,\qquad \forall \theta\in C_4.
\]
The proof relies on the permutation equivariance of FiLM and Transformer blocks under grid rotations, together with the equivariance of the \(G\)-convolutional upsampler [2508.14197].

An important point is that this formal guarantee concerns **rotation equivariance**, not reflection equivariance. Reflection detection is handled by the same architecture, but the theorem in the paper addresses cyclic rotations.

## 4. Semantic-Aware Prompt Grouping

SAPG is the method’s main language-side contribution. Rather than using abstract prompts such as “reflection axis” or “symmetry axes in the image” alone, CLIPSym constructs prompts by concatenating frequent object names extracted from the dataset. A prompt takes the form
\[
t_m=\text{``[obj}_{m_1}\text{] [obj}_{m_2}\text{] }\cdots\text{ [obj}_{m_K}\text{]''}.
\]
For example, with \(K=3\), a prompt may be “apple cloud table” [2508.14197].

The object vocabulary is derived using **Grounded-SAM**, which extracts the **2081 frequent object classes from DENDI**. The prompt set is fixed across images, but the CLIP text encoder is fine-tuned during training, so the text embeddings remain task-adaptive [2508.14197].

SAPG has two grouping levels. First, multiple object words are grouped within each prompt. Second, multiple prompts are used jointly, with learned aggregation weights \(\alpha_t\). The paper reports a search over
\[
M\in\{1,10,25,50\},\qquad K\in\{1,4,8,16,32\},
\]
and finds the best setting to be
\[
M=25,\qquad K=4
\]
for both reflection and rotation experiments [2508.14197].

The DENDI reflection ablation illustrates the effect of prompt design:

| Prompting strategy | F1 |
|---|---:|
| “reflection axis” | 64.4 |
| “symmetry axes in the image” | 64.8 |
| single prompt of frequent object classes (\(K=25\)) | 65.8 |
| multi-prompt \(M=25,K=1\) | 65.3 |
| multi-prompt \(M=25,K=4\) | 66.5 |
| multi-prompt \(M=25,K=16\) | 65.9 |

These results indicate that grouped frequent-object prompts outperform abstract symmetry phrases, and that increasing prompt count or prompt length beyond the \(M=25,K=4\) regime does not continue to improve performance [2508.14197].

The paper’s interpretation is that frequent-object prompts provide stronger initialization in CLIP’s language space because they are closer to CLIP’s pre-training distribution. This suggests that SAPG does not encode symmetry explicitly in language; rather, it exploits object semantics statistically associated with symmetric structure.

## 5. Optimization, datasets, and evaluation protocol

Training uses a pixelwise \(\alpha\)-focal loss:
\[
L_{\text{focal}}(I)=\sum_{x,y} -\alpha'_{S_{xy}} (1-\hat{S}'_{Ixy})^\lambda \log(\hat{S}'_{Ixy}),
\]
where
\[
\hat{S}'_{Ixy}=
\begin{cases}
\hat{S}_{Ixy}, & \text{if } S_{xy}=1,\\
1-\hat{S}_{Ixy}, & \text{otherwise,}
\end{cases}
\qquad
\alpha'_{S_{xy}}=
\begin{cases}
\alpha, & \text{if } S_{xy}=1,\\
1-\alpha, & \text{otherwise.}
\end{cases}
\]
The balancing coefficient is \(\alpha=0.85\) for reflection and \(\alpha=0.95\) for rotation, with \(\lambda=2.0\) in both tasks [2508.14197].

Both the CLIP image encoder and the CLIP text encoder are fine-tuned. The optimizer is **Adam**, training runs for **500** epochs with batch size **16**, and the initial learning rate is chosen from
\[
\{10^{-3},10^{-4},5\times10^{-5},10^{-5},5\times10^{-6},10^{-6}\},
\]
with **\(10^{-5}\)** selected. Reflection detection uses an exponential-decay schedule with decay rate **0.1**, whereas rotation detection uses a constant learning rate. Training augmentation includes random rotations by multiples of \(90^\circ\), random rotations in \([-15^\circ,15^\circ]\), and color jittering [2508.14197].

The evaluation uses three standard symmetry datasets. **DENDI** contains both reflection and rotation subsets; the reflection subset has **2493** images with splits **1750/374/369** for train/val/test, and the rotation subset has **2079** images with splits **1459/313/307**. **SDRW** is used for reflection only, with **51** train and **70** test images. **LDRS** contains **1110/127/240** train/val/test images [2508.14197].

The principal metric is
\[
\mathrm{F1}=\max_\tau
\left(
\frac{2\cdot \mathrm{precision}_\tau\cdot \mathrm{recall}_\tau}
{\mathrm{precision}_\tau+\mathrm{recall}_\tau}
\right),
\]
computed by thresholding predicted heatmaps against binary ground-truth maps. The paper also reports a **Robustness-score**, defined as F1 under transformed inputs, and a **Consistency-score**
\[
\mathrm{Consistency}=
\frac{1}{|D|}\sum_{I\in D}\mathbb{E}_{r}
\left[
\mathrm{CE}\big(r(\hat{S}_I),\hat{S}_{r(I)}\big)
\right],
\]
where lower is better [2508.14197].

## 6. Empirical performance, ablations, and significance

On DENDI, CLIPSym with the equivariant decoder and CLIP pre-training achieves
\[
66.5 \pm 0.2
\]
for reflection and
\[
25.1 \pm 0.1
\]
for rotation. The reproduced EquiSym baseline reports
\[
61.7 \pm 0.6 / 22.0 \pm 0.7,
\]
while the originally reported EquiSym numbers are **64.5 / 22.5**. The paper accordingly reports gains of **+4.8 / +3.1** over the reproduced baseline and **+2.0 / +2.6** over the originally reported EquiSym figures [2508.14197].

On reflection-only benchmarks, CLIPSym reports
\[
51.8 \pm 0.3
\]
on SDRW,
\[
39.5 \pm 0.1
\]
on LDRS, and
\[
42.8 \pm 0.1
\]
on the Mixed benchmark, exceeding EquiSym’s **48.2 / 37.7 / 41.1** in the same settings [2508.14197].

The ablations identify three main sources of improvement. First, **CLIP pre-training** is decisive: CLIPSym trained from scratch attains only **32.1 / 4.7** on DENDI reflection/rotation, whereas the CLIP-pretrained image-only variant reaches **63.7 / 17.7** [2508.14197]. Second, **language helps beyond image-only CLIP**: adding SAPG and text conditioning raises DENDI performance from **63.7 / 17.7** to **66.5 / 25.1**, with particularly large gains in rotation detection [2508.14197]. Third, the **equivariant decoder** contributes beyond a non-equivariant alternative, improving DENDI from **62.9 / 24.2** to **66.5 / 25.1** and also improving reflection scores on SDRW, LDRS, and Mixed [2508.14197].

The trainability ablation shows that fine-tuning the image encoder matters most, but text-side adaptation still contributes. On DENDI reflection, the four configurations yield **59.4** with both encoders frozen, **58.9** with text trainable and image frozen, **65.3** with image trainable and text frozen, and **66.5** with both trainable [2508.14197].

Robustness analysis further separates CLIP feature quality from architectural equivariance. On DENDI reflection under random rotations, CLIPSym non-equivariant achieves robustness **58.3** and consistency **0.093**, while CLIPSym equivariant reaches **59.7** and **0.082**; both outperform EquiSym’s **57.1** and **0.244**. This suggests that CLIP pre-training contributes practical rotation robustness even before the explicit equivariant decoder is applied [2508.14197].

A plausible implication is that CLIPSym’s gains arise from a specific division of labor: CLIP supplies semantically enriched and transformation-tolerant patch features, SAPG adds object-level priors that bias attention toward symmetric structures, and the \(G\)-convolutional decoder restores a geometric inductive bias that CLIP alone does not provide.

## 7. Interpretation, limitations, and position within symmetry detection

CLIPSym is notable because it frames symmetry detection as a problem at the intersection of geometry and semantics. The method does not claim that symmetry is linguistic; instead, it exploits the fact that large-scale caption corpora often encode shape- and object-related regularities. The paper reports that around **10%** of captions in CLIP-scale corpora contain shape or symmetry cues such as “rectangle,” “circle,” and “oval,” and provides frequencies for related words including “ring,” “line,” “arc,” “ball,” and “square” [2508.14197]. This suggests that language supervision can act as an indirect prior over likely symmetric entities.

The method’s limitations are equally clear. SAPG is heuristic: prompt construction is based on frequent-object extraction and hyperparameter search rather than a formal optimization principle. The decoder has a formal **rotation-equivariance** result, but no analogous theorem for reflections. The output remains a dense heatmap rather than a structured symbolic description of symmetry, so CLIPSym does not directly infer axis parameters, symmetry order, or decomposition into symmetric parts. Performance is also highly dependent on large-scale pre-training: the from-scratch ablation shows that the architecture alone is insufficient under current dataset sizes [2508.14197].

The paper additionally points to dataset quality as a substantial source of failure. The qualitative error analysis attributes some missed detections to annotation inconsistency, ambiguous rotation centers, and missing labels for objects that appear symmetric to human observers [2508.14197]. This suggests that remaining error is not solely architectural.

Within the broader symmetry-detection literature, CLIPSym can be understood as a multimodal extension of dense equivariant detection: it preserves the dense heatmap formalism and equivariant decoding tradition while importing CLIP-scale semantic priors through prompt-conditioned feature modulation. Its core claim is therefore narrower than a general statement about language-guided geometry. It shows, more specifically, that pretrained vision-language representations can improve symmetry detection when paired with a decoder that respects geometric structure and a prompting scheme that avoids overly abstract symmetry text [2508.14197].

Source: https://www.emergentmind.com/topics/clipsym