---
title: Patch-Alignment Loss (PAL)
url: https://www.emergentmind.com/topics/patch-alignment-loss-pal
type: topic
---

# Patch-Alignment Loss (PAL)

Patch-Alignment Loss (PAL) denotes a family of objectives that align representations at the level of patches, local crops, spatial tokens, or overlapping local views rather than only at the image, caption, or scene level. In recent arXiv literature, the term spans several distinct constructions: a CLIP-derived patch-attended compatibility for open-vocabulary segmentation, a cosine-distance loss over cross-attention-weighted real and synthetic patch descriptors for Bengali captioning, a symmetric InfoNCE objective for vision–tactile local matching, and a $2$-norm rigid-alignment error for overlapping geometric patches [2212.04994] [2509.18369] [2606.12069] [2303.11620]. This suggests that PAL is not a single canonical loss, but a recurring design principle in which locality is made explicit in the alignment objective.

## 1. Scope of the term

The phrase “Patch-Alignment Loss” is used across multiple research areas, but the aligned objects and the optimization targets differ substantially.

| Setting | Aligned units | PAL form |
|---|---|---|
| Open-vocabulary vision–language learning | ViT patch tokens and a text CLS embedding | Modified CLIP compatibility inside symmetric InfoNCE |
| Bengali captioning | Cross-attention-weighted real and synthetic patch descriptors | Cosine-distance between pooled descriptors |
| Vision–tactile representation learning | Paired local vision and tactile crops | Symmetric InfoNCE |
| Rigid patch registration | Overlapping local geometric views | Sum of squared Frobenius residuals |

A common misconception is that PAL always denotes a contrastive loss. That is inaccurate. In the Bengali-captioning setting, PAL is defined as $1-\cos(r,\tilde r)$ after text-conditioned pooling; in rigid alignment it is the quadratic error $\sum_{(i,j)\in E}\|R_iY_{ij}-R_jY_{ji}\|_F^2$; and in vision–tactile learning it is a symmetric InfoNCE over paired local embeddings [2509.18369] [2303.11620] [2606.12069]. Another misconception is that patch alignment necessarily requires dense supervision. The CLIP-based segmentation formulation explicitly enables open-vocabulary zero-shot segmentation “without requiring any segmentation annotations during training,” while the captioning and vision–tactile variants use other forms of locality-aware supervision [2212.04994] [2509.18369].

## 2. Patch alignment in CLIP-style vision–language models

In "Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning" [2212.04994], the patch-alignment mechanism is introduced as Patch Aligned Contrastive Learning (PACL), a modification of CLIP’s compatibility function. Standard CLIP uses normalized image and text embeddings,
\[
v_i=\frac{e_v(f_v(x_i))}{\|e_v(f_v(x_i))\|}, \qquad
t_i=\frac{e_t(f_t(y_i))}{\|e_t(f_t(y_i))\|},
\]
with compatibility
\[
\phi_{\rm CLIP}(x_i,y_j)=v_i^\top t_j,
\]
and the symmetric InfoNCE objective
\[
\mathcal{L}_{\rm CLIP}
=
-\frac{1}{2k}\sum_{i=1}^k
\left[
\log\frac{\exp(\phi(x_i,y_i)/\tau)}{\sum_{j=1}^k\exp(\phi(x_i,y_j)/\tau)}
+
\log\frac{\exp(\phi(x_i,y_i)/\tau)}{\sum_{j=1}^k\exp(\phi(x_j,y_i)/\tau)}
\right].
\]

PACL replaces the single-token vision encoder by a patch-level encoder
\[
\hat f_v:\mathbb{R}^{C\times H\times W}\to\mathbb{R}^{T\times D_v},
\]
freezes $\hat f_v$ and the text encoder $f_t$, and trains only a small vision embedder
\[
\hat e_v:\mathbb{R}^{D_v}\to\mathbb{R}^{D}.
\]
For each patch $p$, it computes unnormalized patch–text similarities
\[
s_p(x,y)=\bigl(\hat e_v(\hat f_v(x))_{p}\bigr)^\top t,
\]
then softmax-normalizes them across patches to obtain
\[
a_p(x,y)=\frac{\exp(s_p(x,y)/\tau)}{\sum_{q=1}^T\exp(s_q(x,y)/\tau)}.
\]
These weights define a text-conditioned patch aggregate,
\[
\hat v(x,y)=\sum_{p=1}^T a_p(x,y)\,\hat e_v(\hat f_v(x))_{p},
\]
and the scalar compatibility becomes
\[
\hat\phi(x,y)=
\frac{\hat v(x,y)}{\|\hat v(x,y)\|}
\cdot
\frac{e_t(f_t(y))}{\|e_t(f_t(y))\|}.
\]
Replacing $\phi$ by $\hat\phi$ in InfoNCE yields $\mathcal{L}_{\rm PACL}$.

The architectural point is that patch weights $a_p(x,y)$ act as a cross-modal attention mechanism over vision patches, while the text side remains a single CLS embedding. The paper states that only $\hat e_v$ is trained; the vision encoder $\hat f_v$, text encoder $f_t$, and text embedder $e_t$ remain frozen. The formulation is compatible with CLIP ViT-B/16 and ViT-L/14, and even with a DINO-trained ViT vision encoder. A “stride trick” at inference reduces patch stride, for example from $16\times16$ to $4\times4$, to obtain a finer patch grid without retraining.

The central empirical claim is that aligning patches rather than only CLS tokens unlocks open-vocabulary zero-shot segmentation. On Pascal VOC, per-patch zero-shot classification accuracy rises from $\sim 52.5\%$ to above $95\%$ after PACL training; the detailed figures are $52.5\%\to96.5\%$ for CLIP ViT-B/16 and $27.9\%\to95.3\%$ for CLIP ViT-L/14. Zero-shot semantic segmentation mIoU reaches $72.3\%$ on Pascal VOC-20, $50.1\%$ on Pascal Context-59, $38.8\%$ on COCO-Stuff-171, and $31.4\%$ on ADE20K-150, with no masks or pixel labels used at train time. The same patch-alignment mechanism also improves zero-shot image classification, including ImageNet top-1 from $68.7\to73.6$ for ViT-B/16 and from $76.0\to78.2$ for ViT-L/14.

## 3. Cross-attention-guided PAL for Bengali captioning

In "Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning" [2509.18369], PAL is defined on matched triplets $(x,\tilde x,y)$, where $x$ is a real image, $\tilde x$ is a synthetic counterpart generated from the same Bengali caption $y=(y_1,\dots,y_T)$, and a frozen MaxViT encoder provides a feature map $F(x)\in\mathbb{R}^{C\times H\times W}$ with $S=H\cdot W$ spatial locations. After reshaping and projecting,
\[
E(x)=\mathrm{LN}\bigl((F(x)\!\to\![S\times C])\,W_p\bigr)\in\mathbb{R}^{S\times D},
\]
and similarly for $E(\tilde x)$.

The distinctive component is that decoder cross-attention determines which patches matter. During autoregressive decoding with mBART-50, cross-attention maps $\mathcal A^{(\ell,h)}\in\mathbb{R}^{T\times S}$ are averaged across the last $K$ layers and over heads,
\[
\bar A=\frac1K\sum_{\ell=L-K+1}^{L}\mathrm{mean}_h\bigl(\mathcal A^{(\ell,h)}\bigr)\in\mathbb{R}^{T\times S},
\]
then summed over time and passed through TopKSoftmax,
\[
w=\mathrm{TopKSoftmax}\Bigl(\sum_{t=1}^{T}\bar A_{t,\cdot};\tau,\rho\Bigr)\in\Delta^{S-1}.
\]
TopKSoftmax retains only the minimal set of indices whose cumulative pre-softmax mass is $\rho$, then renormalizes. Using these weights, the model forms pooled, text-conditioned descriptors
\[
r=\sum_{s=1}^{S}w_sE_s(x), \qquad
\tilde r=\sum_{s=1}^{S}w_sE_s(\tilde x),
\]
and the Patch-Alignment Loss is
\[
\mathcal L_{\mathrm{PAL}(x,\tilde x,y)}
=
1-\cos(r,\tilde r)
=
1-\frac{\langle r,\tilde r\rangle}{\|r\|\|\tilde r\|}.
\]

PAL is embedded in a tri-loss objective,
\[
\mathcal L
=
\mathcal L_{\mathrm{CE}(x,y)}
+
\lambda_{\mathrm{PAL}}\mathcal L_{\mathrm{PAL}(x,\tilde x,y)}
+
\alpha\,\mathcal L_{\mathrm{InfoNCE}}
+
\beta\,\mathcal L_{\mathrm{OT}}.
\]
Here CE trains the decoder on real captions, InfoNCE enforces global real–synthetic separation, and Sinkhorn-based OT encourages balanced fine-grained patch correspondence. The paper specifies default hyperparameters $\lambda_{\mathrm{PAL}}=0.5$, $\alpha=0.3$, $\beta=0.5$, attention temperature $\tau=1.0$, top-mass $\rho=0.5$, last-$K=2$ layers, InfoNCE temperature $t=0.07$, OT regularizer $\epsilon=0.05$, and $30$ Sinkhorn iterations. Gradients flow only through the linear+LN projection and mBART-50; MaxViT remains frozen.

The empirical ablations isolate PAL’s role. On Flickr30k-1k, CE (Real only) yields BLEU-4 $5.80$, METEOR $24.96$, and BERT-F1 $68.38$; CE + InfoNCE + OT yields $7.52$, $26.22$, and $70.21$; PAL (only) yields $10.19$, $27.29$, and $70.97$; and PAL + InfoNCE + OT reaches $12.29$, $27.98$, and $71.20$. On MSCOCO-1k, the full pipeline reports BLEU-4 $12.00$, METEOR $28.14$, and BERTScore-F1 $75.40$. Grounding analysis shows the real–synthetic centroid distance of attended-patch embeddings falling from $0.361\to0.213$ and MMD-RBF from $0.013\to0.002$, while background patches remain dispersed. This supports the paper’s claim that PAL is the core driver of text-conditioned grounding.

## 4. Vision–tactile PAL and local cross-modal alignment

In "Tac-DINO: Learning Vision-Tactile Features with Patch Alignment" [2606.12069], PAL is a symmetric InfoNCE objective over paired local vision and tactile patches. The data unit is a contact-aligned pair: $v_i=\mathrm{Crop}_\ell(I_i;p_i)\in\mathbb{R}^{3\times h\times h}$ is a local RGB crop around the contact pixel $p_i$, and $t_i\in\mathbb{R}^{3\times h\times h}$ is the corresponding tactile image recorded by the GelSight Mini. A ViT-Large from DINOv2 encodes each modality,
\[
E_v:\mathbb{R}^{3\times h\times h}\to\mathbb{R}^{d}, \qquad
E_t:\mathbb{R}^{3\times h\times h}\to\mathbb{R}^{d},
\]
with $d=1024$, producing embeddings $f_{v_i}=E_v(v_i)$ and $f_{t_i}=E_t(t_i)$. Cosine similarity is
\[
\mathrm{sim}(u,v)=\frac{u^\top v}{\|u\|_2\|v\|_2}.
\]

The loss is
\[
L_{\mathrm{PAL}}=L_{v\to t}+L_{t\to v},
\]
where
\[
L_{v\to t}
=
-\frac1N\sum_{i=1}^{N}
\log
\frac{\exp(\mathrm{sim}(f_{v_i},f_{t_i})/\tau)}
{\sum_{j=1}^{N}\exp(\mathrm{sim}(f_{v_i},f_{t_j})/\tau)},
\]
and
\[
L_{t\to v}
=
-\frac1N\sum_{i=1}^{N}
\log
\frac{\exp(\mathrm{sim}(f_{t_i},f_{v_i})/\tau)}
{\sum_{j=1}^{N}\exp(\mathrm{sim}(f_{t_i},f_{v_j})/\tau)}.
\]
Typical implementation uses $\tau=0.1$. This loss is combined with DINOv2-style self-distillation on global crops,
\[
L=L_{\mathrm{DINO\_global}}+\lambda_{\mathrm{PAL}}L_{\mathrm{PAL}},
\]
with $\lambda_{\mathrm{PAL}}=1.0$. Reported optimization settings include batch size per GPU $32$ on $4$ GPUs for effective batch $128$, learning rate $1\times10^{-4}$ with $40$-epoch linear warmup and cosine decay over $200$ total epochs, weight decay $0.05$, gradient clipping $1.0$, and AdamW.

The formulation depends on the Touch3D dataset’s per-contact supervision. The dataset contains $505$ unique real-world objects across $6$ household/office categories, $20\,025$ discrete tactile contacts, average $\sim 40$ contacts per object, high-resolution $3$D meshes with $0.2$ mm point resolution, RGB $(224\times224)$ and depth of the global scene, multi-view renderings for contact-pixel annotation, and $18$ material labels, with each object having up to $4$ materials. The patch-sampling pipeline labels the contact location on a high-resolution scan, renders multi-view RGB to identify visible views, randomly samples contact indices during training, loads $(I_i,p_i,t_i)$, generates $v_i$, encodes $(f_{v_i},f_{t_i})$, and computes $L_{\mathrm{PAL}}$ over the minibatch.

On the Vis-Tac Holographic Matching Benchmark, PAL yields substantial local-to-global retrieval gains. For tactile-side retrieval, Patch Tac gives $21.05\%$ top-1, Patch Tac + PAL gives $34.23\%$, Patch Tac + DSCMR gives $30.15\%$, and Patch Tac + DAR gives $48.30\%$. For vision-side retrieval, Patch Vis gives $52.63\%$, Patch Vis + PAL gives $69.50\%$, and Patch Vis + DAR gives $57.32\%$. Vision–tactile early fusion improves from $51.69\%$ without PAL to $55.96\%$ with PAL. The paper further states that, on the held-out OBJECTFOLDER REAL test set, similar relative gains are observed at $+10$–$15$ percentage points over baselines, PAL improves material mAP by $\sim 10$ percentage points, and reduces Chamfer Distance marginally in geometry matching. The ablations also report that $\tau=0.1$ is best over a sweep of $0.05$–$0.2$, that $\lambda_{\mathrm{PAL}}=1$ is a plateau point with slight overfitting above $1.5$, and that patch-level alignment yields $+20$–$35$ percentage points over entire-image contrastive alignment.

## 5. PAL in rigid alignment and patch-based geometry

In "Non-degenerate Rigid Alignment in a Patch Framework" [2303.11620], PAL is not a neural representation loss but a geometric objective over overlapping local views of a point cloud. Let $\mathcal G=(V,E)$ be the patch-overlap graph with $V=\{1,\dots,N\}$. For each overlap $(i,j)\in E$, let $Y_{ij}$ and $Y_{ji}$ be the centered local coordinates of the $n_{ij}$ shared points in patches $i$ and $j$, and let $R_i\in O(d)$ be the orthonormal map assigned to patch $i$. The Patch-Alignment Loss is
\[
\mathrm{PAL}(R)=\sum_{(i,j)\in E}\bigl\|R_iY_{ij}-R_jY_{ji}\bigr\|_F^2.
\]
This is the canonical $2$-norm-based alignment error.

The paper rewrites the loss as a quadratic form. Stacking the rotations into
\[
Q=
\begin{bmatrix}
R_1\\
\vdots\\
R_N
\end{bmatrix}
\in\mathbb{R}^{dN\times d},
\]
and defining a sparse patch-incidence block matrix $B$, one obtains
\[
\mathrm{PAL}(R)=\|BQ\|_F^2=\mathrm{Tr}(Q^\top C Q),
\qquad C=B^\top B.
\]
The alignment matrix $C$ has blocks
\[
C_{ii}=\sum_{j:(i,j)\in E}Y_{ij}Y_{ij}^\top,\qquad
C_{ij}=-Y_{ij}Y_{ji}^\top\ \ (i\neq j),
\]
and $C\succeq 0$.

The central theoretical notion is non-degeneracy of a perfect alignment. Because $\mathrm{PAL}(GR^*)=\mathrm{PAL}(R^*)$ for any global $G\in O(d)$, a zero-loss solution always has a $d(d-1)/2$-dimensional global-rotation symmetry. A perfect alignment is non-degenerate when the Hessian restricted to the tangent of $O(d)^N$ is positive definite on the complement of those symmetry directions. Equivalently, for $C^*=B^\top B$ evaluated at $R^*$, the kernel must be exactly
\[
\{\,Q^*\Omega:\Omega^\top=-\Omega\in\mathbb{R}^{d\times d}\,\},
\]
and all remaining eigenvalues must be strictly positive.

Because $C^*$ is explicit, the paper gives a polynomial-time non-degeneracy test based on the eigenvalues of $C^*$. If $m=\tfrac{d(d-1)}2$ and
\[
0=\lambda_1=\cdots=\lambda_m\le \lambda_{m+1}\le \cdots\le \lambda_{dN},
\]
then the alignment is declared non-degenerate when $\lambda_{m+1}>0$ above tolerance and the first $m$ eigenvalues are zero within tolerance. The reported dense eigendecomposition cost is $O((dN)^3)$, with sparse methods potentially faster.

The optimization analysis is formulated on the manifold
\[
\mathcal M=O(d)\times\cdots\times O(d)\subset\mathbb{R}^{dN\times d}.
\]
For
\[
f(R)=\mathrm{Tr}(R^\top C R),
\]
the Euclidean gradient is $\nabla f(R)=2CR$, and the Riemannian gradient is
\[
\operatorname{grad}f(R)=2CR-R\,\skew(R^\top C R),
\]
with blockwise tangent projection and a blockwise QR-based retraction. If $R^*$ is a non-degenerate critical point, then in a small neighborhood of $R^*$ the function satisfies a Riemannian strong-convexity condition. With $\mu_{\min}$ the smallest nonzero eigenvalue of $C^*$ and $L$ a Lipschitz constant for the gradient, any fixed stepsize $0<\eta<2/L$ yields local linear convergence,
\[
\mathrm{dist}(R_k,R^*)\le
\bigl(1-\eta\mu_{\min}+O(\eta^2)\bigr)^k
\mathrm{dist}(R_0,R^*).
\]

The paper also gives a noise-stability statement. If the perturbation of the alignment matrix obeys
\[
\|\Delta\|_2<\tfrac12\,\mu_{\min},
\]
then the noisy problem remains non-degenerate, the global minimizer $\widetilde R$ obeys
\[
\mathrm{dist}(\widetilde R,R^*)\le
C\,\frac{\|\Delta\|_2}{\mu_{\min}},
\]
for an absolute constant $C$, and Riemannian gradient descent initialized near $R^*$ converges linearly to $\widetilde R$. In the noiseless case, the paper further shows that non-degenerate perfect alignment is equivalent to infinitesimal rigidity of the patch-overlap graph for generic point positions, with rank
\[
dN-\tfrac{d(d-1)}2,
\]
and that global rigidity yields uniqueness up to a single global orthogonal map.

## 6. Comparative interpretation

Across these formulations, PAL always elevates local structure to the level of the training objective, but the mechanism by which locality is specified differs. In PACL, the relevant patches are induced by similarities between patch embeddings and a text CLS token [2212.04994]. In Bengali captioning, locality is defined by decoder cross-attention, averaged across layers and heads and pruned by TopKSoftmax [2509.18369]. In vision–tactile learning, locality is provided by per-contact annotations that bind a tactile reading to a specific vision crop [2606.12069]. In rigid alignment, locality is encoded by the overlap graph and shared coordinates between patches [2303.11620].

This suggests three broad PAL archetypes. The first is **text-conditioned pooling**, in which a patch-weighted descriptor is built before computing similarity, as in PACL and Bengali captioning. The second is **pairwise local contrastive matching**, in which paired patches directly enter a symmetric InfoNCE loss, as in the vision–tactile formulation. The third is **quadratic consistency over overlaps**, in which patch alignment is a rigid-registration objective rather than a representation-learning loss.

The main significance of this family of objectives is methodological rather than notational. Patch alignment repeatedly appears when global embeddings are too coarse for the target task: CLIP CLS alignment is insufficient for dense semantic segmentation; caption-level fluency does not ensure that a Bengali decoder attends to the correct objects; whole-image vision–tactile matching ignores the locality of contact; and rigid patch registration requires overlap-wise consistency rather than only global fit. A plausible implication is that PAL-like objectives become most useful when the supervision signal is semantically or physically local even if the downstream prediction is global.

A second implication is that PAL does not prescribe a single supervision regime. One formulation achieves zero-shot segmentation without segmentation labels, another exploits real–synthetic pairs plus caption supervision, another uses contact-level pairing from a dedicated dataset, and another is entirely geometric. Accordingly, “PAL” is best treated as a patch-level alignment paradigm whose exact mathematical form depends on the structure of correspondence available in the problem.

Source: https://www.emergentmind.com/topics/patch-alignment-loss-pal