---
title: Transformer over Similarity Maps
url: https://www.emergentmind.com/topics/transformer-over-similarity-maps
type: topic
---

# Transformer over Similarity Maps

“Transformer over similarity maps” (*Editor’s term*) denotes a family of formulations in which the transformer’s primary computational object is not only a raw token sequence or feature map, but an explicit similarity structure: Siamese cross-correlation maps, pairwise image-comparison tensors, similarity-derived attention masks, modality-alignment matrices, graph-embedding distance matrices, or layerwise and tokenwise representational similarity maps. The pattern is most explicit in HiFT for aerial tracking, where a transformer operates directly on hierarchical similarity maps produced by a Siamese tracker, but closely related constructions recur in pairwise image similarity, dense detection, multimodal fusion, text similarity explanation, graph retrieval, and measure-theoretic analyses of self-attention [2108.00202], [2407.16981], [2409.19323], [2405.05760], [2208.06612], [2506.15907], [2509.25611].

## 1. Conceptual scope and recurring formulations

A useful synthesis is to distinguish three broad uses of similarity maps. In one use, similarity maps are the **direct transformer input**. HiFT is the canonical example: cross-correlation maps from multiple Siamese backbone layers are flattened into tokens and passed to an encoder–decoder transformer [2108.00202]. In a second use, a transformer computes a **pairwise similarity score**, and internal attention or saliency structures are then exposed as similarity explanation maps, as in CEViT and BERT-based text similarity interpretation [2407.16981], [2208.06612]. In a third use, **externally computed similarity maps** modulate attention or feed-forward blocks, as in FishViT and SG-MFT [2409.19323], [2405.05760].

A further extension appears when similarity maps are treated as **analytical objects** rather than forward-pass inputs. TGO-II constructs layer–layer similarity maps with CKA and SVCCA, token–token similarity maps with covariance, and manifold-complexity maps with TwoNN-ID in order to study ViT training dynamics [2607.02386]. At the most abstract level, the measure-theoretic formulation of transformers models self-attention as an in-context map that pushes a measure forward under a similarity-dependent transport rule [2509.25611].

| Setting | Similarity object | Transformer role |
|---|---|---|
| HiFT | Hierarchical Siamese correlation maps | Fuses multi-level spatial and semantic cues |
| CEViT | Pair-conditioned attention masks | Predicts same/different similarity and explanations |
| FishViT | Similarity-derived aggregation and thresholds | Denoises attention in dense scenes |
| SG-MFT | Modality-wise and element-wise similarity maps | Gates self-/cross-attention and FFN interaction |
| BTI | Activation and saliency maps over token embeddings | Explains paragraph similarity via salient word pairs |

This suggests that “transformer over similarity maps” is not a single architecture, but a design principle: move computation from raw appearance or raw tokens toward a representation in which similarity is already explicit, then let attention refine, fuse, or explain that representation.

## 2. Correlation-space transformers in Siamese tracking

HiFT places the idea in its most literal form. In Siamese tracking, a template patch \(\mathbf{Z}\) and a search patch \(\mathbf{X}\) are encoded by a shared backbone, and a layerwise similarity map is computed by cross-correlation,
\[
\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).
\]
HiFT argues that relying only on the last-layer similarity map degrades localization accuracy in complex scenarios, while treating multiple maps separately introduces intractable computations for aerial mobile platforms. Its solution is to transform the similarity maps themselves rather than the raw backbone features [2108.00202].

For AlexNet layers \(i\in\{3,4,5\}\), HiFT forms
\[
\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},
\]
so each spatial position of each similarity map becomes a token. The encoder combines shallow maps through
\[
\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),
\]
and computes attention as
\[
\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').
\]
A modulation layer then uses \(\mathbf{M}_4'\), global average pooling, a feed-forward network, and a learnable scalar \(\gamma_1\) to reweight the encoded shallow similarity representation. The decoder operates on the deep similarity map \(\mathbf{M}_5\) without explicit positional encoding on \(\mathbf{M}_5\); the ablation reported in the paper shows that direct positional encoding there hurts performance.

Architecturally, the feature transformer has **1 encoder layer** and **2 decoder layers**, and the full system remains end-to-end: the decoded similarity representation is reshaped back to a spatial tensor and sent to classification and regression heads with
\[
L_{\text{overall}}=\lambda_1L_{\text{cls1}}+\lambda_2L_{\text{cls2}}+\lambda_3L_{\text{loc}}.
\]
The empirical result is not merely conceptual. On UAV20L, the baseline reports precision \(0.611\) and success \(0.463\), whereas Baseline + HFT reports \(0.763\) and \(0.566\). Runtime is reported at approximately \(130\) FPS on TITAN RTX and approximately \(31.2\) FPS on NVIDIA AGX Xavier without TensorRT [2108.00202].

HiFT is therefore the clearest instance in which a transformer is placed **after similarity computation but before prediction**. The transformer does not discover similarity from scratch; it reorganizes already computed match evidence across levels and positions.

## 3. Pairwise similarity prediction and explanation

CEViT reinterprets a Vision Transformer as a learned similarity function over image pairs rather than a single-image classifier. Two images \((x_q,x_r)\) are concatenated along the channel dimension,
\[
\tilde{x}=\mathrm{concat}_{\mathrm{channel}}(x_q,x_r)\in\mathbb{R}^{h\times w\times 2c},
\]
patchified, and processed by a standard ViT encoder with **6 encoder layers**. The output head is changed from multi-class classification to a scalar,
\[
s=\sigma\big(\mathrm{MLP}(\mathrm{CLS}_{\mathrm{final}})\big),
\]
interpreted as \(P(\text{same class}\mid x_q,x_r)\). Internally, CLS-to-patch attention is reshaped to an \(N\times N\) grid, upsampled, and visualized as a similarity explanation map. In the reported k-NN setup with \(K=15\), CEViT + k-NN attains \(99.0\%\) on MNIST, close to a ViT classifier at \(99.1\%\), while its distraction scores are positive at approximately \(0.026\)–\(0.037\), unlike vanilla ViT attention scores, which are near \(0\) or slightly negative [2407.16981].

The text-similarity analogue is provided by the BERT-based interpretation framework for paragraph similarity. There the paragraph embedding is
\[
F_p=\frac{1}{q}\sum_{i=2}^{q+1} B(p)_i,
\]
and similarity is cosine similarity between two such embeddings. The method then differentiates this scalar similarity with respect to the input embedding activations \(E(I^p)\), producing token saliency, aggregates WordPiece tokens to words, and ranks cross-paragraph word pairs by
\[
U(w_i^1,w_j^2):=c^{ij}\cdot s_i^1\cdot s_j^2.
\]
MeanShift clustering on the word-pair scores selects the top explanatory matches. In the reported human evaluation on Fashion and Wine review data, the full method achieves \(4.3\pm0.3\) MOS, compared with \(3.5\pm0.4\) for Integrated Gradients and \(3.6\pm0.5\) for Vanilla Gradients [2208.06612].

These systems use similarity maps differently from HiFT. CEViT computes similarity directly from a pair-conditioned transformer input and then reads explanations from attention. The BERT-based method computes a global similarity score first and then lifts it back into token- and word-level saliency maps. In both cases, however, similarity is not a downstream byproduct of classification; it is the central supervised quantity.

## 4. Similarity-guided attention and fusion in dense vision and multimodal learning

FishViT makes similarity-awareness explicit inside the attention mechanism. Starting from factorized attention, it defines
\[
A=\mathrm{Softmax}(K)^\top V,
\]
treats this as a similarity-derived global aggregation, and computes adaptive channel-wise thresholds
\[
\tau_c=\alpha_c\cdot \mathrm{average}|A_{i,j,c}|.
\]
The encoder uses three parallel branches over ResNet-18 feature levels \(l\in\{3,4,5\}\), each with multi-level patch embedding, pooling positional encoding
\[
\mathcal{M}_P=\mathrm{AvgPooling}(\mathcal{M})+\mathcal{M},
\]
and soft-threshold attention. The intended effect is to suppress weak similarities associated with background noise while preserving strong similarities associated with fish contours and discriminative structures. On the reported benchmark, FishViT attains \(\mathrm{AP}_{50}^{val}=94.7\), \(E_{cls}=0.11\), \(E_{loc}=1.65\), \(E_{miss}=0.28\), and \(82.3\) FPS. In ablation, \(\mathrm{AP}_{50}\) rises from \(89.1\) for the baseline to \(92.6\) with STAttention, \(93.2\) with the multi-level encoder, and \(94.7\) for the full model [2409.19323].

SG-MFT uses external similarity maps as explicit control signals for multimodal fusion. From Chinese-CLIP text and image features, it forms coarse modality-wise similarity \(P^m=\mathrm{softmax}(S^m)\) and fine element-wise similarity \(P^e\) from
\[
S^t=F_{\text{text}}(F_{\text{image}})^T.
\]
The similarity-guided interaction module interpolates cross-attention and self-attention according to
\[
X_{\text{image}}=CA\otimes P^m \oplus SA\otimes(1-P^m),
\]
and injects fine-grained cross-modal alignment through
\[
X_{\text{inter}}=P^eX_{\text{image}}^C
\]
inside a similarity-aware feed-forward block. On the Multimedia Weibo Social Media Dataset, the full SG-MFT reaches \(87.29\%\) accuracy, compared with \(86.94\%\) for Baseline + CLIP + SIM and \(85.74\%\) for Baseline + CLIP + SFM [2405.05760].

The commonality between FishViT and SG-MFT is architectural. Neither merely visualizes attention after training. Instead, both alter the transformer computation itself by introducing similarity-dependent gating, shrinkage, or interpolation. In this regime, the similarity map is an operator on attention, not just an interpretive overlay.

## 5. Graph, geometric, and measure-theoretic extensions

In graph-based similarity learning, Pieceformer defines similarity through graph-level embeddings \(\mathbf{g}_i\) and L2 distances,
\[
S_i=\|\mathbf{g}_t-\mathbf{g}_i\|.
\]
Its encoder combines message passing with a Partitioned Graph Transformer using linear attention and METIS-based partitioning. The transformer therefore operates on partition-local node interactions, while the resulting graph embeddings induce a corpus-level similarity matrix \(S_{ij}=\|\mathbf{g}_i-\mathbf{g}_j\|\). The reported partition-size ablation finds the best MAE around \(500\) nodes per partition. Across scales, GIN+PGT achieves the lowest MAE across nearly all settings, with a \(24.9\%\) average MAE reduction over GIN across all scales, and the partitioning case study reports up to \(89\%\) runtime reduction [2506.15907].

TGO-II shifts from architectural usage to geometric analysis. It computes layer–layer similarity with CKA and SVCCA, token–token similarity with token covariance and a coupling metric, and manifold complexity with TwoNN-ID. For ViT-S/16 on ImageNet-100, mean CKA and mean SVCCA decrease throughout training, intrinsic dimensionality increases before stabilizing, and token covariance retains substantial off-diagonal structure. The paper’s interpretation is that representational specialization and manifold expansion emerge together, while strong token interaction structure persists [2607.02386].

At the most abstract level, the measure-theoretic account of transformers defines an in-context map \(G\) and a push-forward measure map
\[
f(\mu)=G(\mu)_\#\mu.
\]
Within this formulation, self-attention appears as a similarity kernel inside \(G\), and the central characterization theorem states that continuous representations of this kind are equivalent to support-preserving maps whose regular part of the Fréchet derivative is uniformly continuous. The same framework further identifies the infinite-depth, mean-field measure-theoretic transformer with a Vlasov flow [2509.25611].

These extensions broaden the meaning of the topic. In Pieceformer, the similarity map is induced downstream by transformer-produced embeddings. In TGO-II, similarity maps become observables of representation geometry. In the measure-theoretic setting, similarity is the kernel that defines the transport field of the transformer itself.

## 6. Design patterns, misconceptions, and limitations

Across the literature, several design patterns recur. One pattern converts task-specific similarity structures into tokens, as in HiFT’s hierarchical cross-correlation maps [2108.00202]. A second computes a scalar similarity score from a pairwise transformer input and then reuses internal structures as explanations, as in CEViT and BERT-based paragraph similarity interpretation [2407.16981], [2208.06612]. A third injects external similarity maps into attention or feed-forward computation, as in FishViT and SG-MFT [2409.19323], [2405.05760]. A fourth uses transformer-produced embeddings to induce a downstream similarity map over a corpus, as in Pieceformer [2506.15907].

A common misconception is that the topic is exhausted by standard \(QK^\top\) attention. The literature is broader. HiFT applies the transformer to correlation maps rather than raw backbone features [2108.00202]. CEViT does not introduce an explicit cross-image attention layer; it fuses the image pair by channel concatenation before patchification [2407.16981]. The BERT-based text-similarity method uses activation and saliency maps to explain a similarity score after the fact [2208.06612]. TGO-II studies similarity maps of representations themselves rather than task-level similarity heads [2607.02386]. A plausible implication is that “transformer over similarity maps” is best understood as a computational stance toward similarity-structured intermediate objects, not as a single attention formula.

The empirical scope remains uneven. CEViT is reported only on MNIST and is explicitly described as preliminary research [2407.16981]. SG-MFT depends on similarity estimates supplied by Chinese-CLIP [2405.05760]. FishViT notes limitations under extreme occlusions, low contrast, and domain shift [2409.19323]. Pieceformer identifies edge-feature integration and adaptive partitioning as open directions [2506.15907]. TGO-II studies one architecture, ViT-S/16, on ImageNet-100 [2607.02386]. The broader field therefore supports the design principle strongly, but its instantiations remain domain-specific.

Taken together, these works establish a coherent technical idea: transformers can operate not only on appearance or token content, but on explicit similarity structures that encode correspondence, alignment, match evidence, or representational geometry. In some systems this produces more efficient multi-level fusion, in others more faithful similarity scoring, more discriminative attention, more interpretable explanations, or a clearer theoretical description of self-attention itself.

Source: https://www.emergentmind.com/topics/transformer-over-similarity-maps