Transformer over Similarity Maps

• Transformer over similarity maps is a design principle where explicit similarity structures, such as Siamese cross-correlation maps, serve as inputs to transformer architectures.
• The approach integrates methods like pair-conditioned attention, similarity-derived thresholds, and external similarity modulation to improve tasks like tracking and multimodal fusion.
• Applications include aerial mobile tracking, text and image similarity explanation, dense detection, graph retrieval, and theoretical analysis of self-attention.

“Transformer over similarity maps” (Editor’s term) denotes a family of formulations in which the transformer’s primary computational object is not only a raw token sequence or feature map, but an explicit similarity structure: Siamese cross-correlation maps, pairwise image-comparison tensors, similarity-derived attention masks, modality-alignment matrices, graph-embedding distance matrices, or layerwise and tokenwise representational similarity maps. The pattern is most explicit in HiFT for aerial tracking, where a transformer operates directly on hierarchical similarity maps produced by a Siamese tracker, but closely related constructions recur in pairwise image similarity, dense detection, multimodal fusion, text similarity explanation, graph retrieval, and measure-theoretic analyses of self-attention (Cao et al., 2021, Zhao et al., 2024, Li et al., 2024, Zhang et al., 2024, Malkiel et al., 2022, Yang et al., 18 Jun 2025, Furuya et al., 30 Sep 2025).

1. Conceptual scope and recurring formulations

A useful synthesis is to distinguish three broad uses of similarity maps. In one use, similarity maps are the direct transformer input. HiFT is the canonical example: cross-correlation maps from multiple Siamese backbone layers are flattened into tokens and passed to an encoder–decoder transformer (Cao et al., 2021). In a second use, a transformer computes a pairwise similarity score, and internal attention or saliency structures are then exposed as similarity explanation maps, as in CEViT and BERT-based text similarity interpretation (Zhao et al., 2024, Malkiel et al., 2022). In a third use, externally computed similarity maps modulate attention or feed-forward blocks, as in FishViT and SG-MFT (Li et al., 2024, Zhang et al., 2024).

A further extension appears when similarity maps are treated as analytical objects rather than forward-pass inputs. TGO-II constructs layer–layer similarity maps with CKA and SVCCA, token–token similarity maps with covariance, and manifold-complexity maps with TwoNN-ID in order to study ViT training dynamics (Kapil et al., 2 Jul 2026). At the most abstract level, the measure-theoretic formulation of transformers models self-attention as an in-context map that pushes a measure forward under a similarity-dependent transport rule (Furuya et al., 30 Sep 2025).

Setting	Similarity object	Transformer role
HiFT	Hierarchical Siamese correlation maps	Fuses multi-level spatial and semantic cues
CEViT	Pair-conditioned attention masks	Predicts same/different similarity and explanations
FishViT	Similarity-derived aggregation and thresholds	Denoises attention in dense scenes
SG-MFT	Modality-wise and element-wise similarity maps	Gates self-/cross-attention and FFN interaction
BTI	Activation and saliency maps over token embeddings	Explains paragraph similarity via salient word pairs

This suggests that “transformer over similarity maps” is not a single architecture, but a design principle: move computation from raw appearance or raw tokens toward a representation in which similarity is already explicit, then let attention refine, fuse, or explain that representation.

2. Correlation-space transformers in Siamese tracking

HiFT places the idea in its most literal form. In Siamese tracking, a template patch $\mathbf{Z}$ and a search patch $\mathbf{X}$ are encoded by a shared backbone, and a layerwise similarity map is computed by cross-correlation,

$$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$

HiFT argues that relying only on the last-layer similarity map degrades localization accuracy in complex scenarios, while treating multiple maps separately introduces intractable computations for aerial mobile platforms. Its solution is to transform the similarity maps themselves rather than the raw backbone features (Cao et al., 2021).

For AlexNet layers $i\in\{3,4,5\}$, HiFT forms

$$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$

so each spatial position of each similarity map becomes a token. The encoder combines shallow maps through

$$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$

and computes attention as

$$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$

A modulation layer then uses $\mathbf{M}_4'$, global average pooling, a feed-forward network, and a learnable scalar $\gamma_1$ to reweight the encoded shallow similarity representation. The decoder operates on the deep similarity map $\mathbf{M}_5$ without explicit positional encoding on $\mathbf{X}$0; the ablation reported in the paper shows that direct positional encoding there hurts performance.

Architecturally, the feature transformer has 1 encoder layer and 2 decoder layers, and the full system remains end-to-end: the decoded similarity representation is reshaped back to a spatial tensor and sent to classification and regression heads with

$\mathbf{X}$1

The empirical result is not merely conceptual. On UAV20L, the baseline reports precision $\mathbf{X}$2 and success $\mathbf{X}$3, whereas Baseline + HFT reports $\mathbf{X}$4 and $\mathbf{X}$5. Runtime is reported at approximately $\mathbf{X}$6 FPS on TITAN RTX and approximately $\mathbf{X}$7 FPS on NVIDIA AGX Xavier without TensorRT (Cao et al., 2021).

HiFT is therefore the clearest instance in which a transformer is placed after similarity computation but before prediction. The transformer does not discover similarity from scratch; it reorganizes already computed match evidence across levels and positions.

3. Pairwise similarity prediction and explanation

CEViT reinterprets a Vision Transformer as a learned similarity function over image pairs rather than a single-image classifier. Two images $\mathbf{X}$8 are concatenated along the channel dimension,

$\mathbf{X}$9

patchified, and processed by a standard ViT encoder with 6 encoder layers. The output head is changed from multi-class classification to a scalar,

$$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$0

interpreted as $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$1. Internally, CLS-to-patch attention is reshaped to an $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$2 grid, upsampled, and visualized as a similarity explanation map. In the reported k-NN setup with $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$3, CEViT + k-NN attains $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$4 on MNIST, close to a ViT classifier at $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$5, while its distraction scores are positive at approximately $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$6–$$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$7, unlike vanilla ViT attention scores, which are near $$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$8 or slightly negative (Zhao et al., 2024).

The text-similarity analogue is provided by the BERT-based interpretation framework for paragraph similarity. There the paragraph embedding is

$$\mathbf{S}_k=\phi_k(\mathbf{Z})\star \phi_k(\mathbf{X}).$$9

and similarity is cosine similarity between two such embeddings. The method then differentiates this scalar similarity with respect to the input embedding activations $i\in\{3,4,5\}$0, producing token saliency, aggregates WordPiece tokens to words, and ranks cross-paragraph word pairs by

$i\in\{3,4,5\}$1

MeanShift clustering on the word-pair scores selects the top explanatory matches. In the reported human evaluation on Fashion and Wine review data, the full method achieves $i\in\{3,4,5\}$2 MOS, compared with $i\in\{3,4,5\}$3 for Integrated Gradients and $i\in\{3,4,5\}$4 for Vanilla Gradients (Malkiel et al., 2022).

These systems use similarity maps differently from HiFT. CEViT computes similarity directly from a pair-conditioned transformer input and then reads explanations from attention. The BERT-based method computes a global similarity score first and then lifts it back into token- and word-level saliency maps. In both cases, however, similarity is not a downstream byproduct of classification; it is the central supervised quantity.

4. Similarity-guided attention and fusion in dense vision and multimodal learning

FishViT makes similarity-awareness explicit inside the attention mechanism. Starting from factorized attention, it defines

$i\in\{3,4,5\}$5

treats this as a similarity-derived global aggregation, and computes adaptive channel-wise thresholds

$i\in\{3,4,5\}$6

The encoder uses three parallel branches over ResNet-18 feature levels $i\in\{3,4,5\}$7, each with multi-level patch embedding, pooling positional encoding

$i\in\{3,4,5\}$8

and soft-threshold attention. The intended effect is to suppress weak similarities associated with background noise while preserving strong similarities associated with fish contours and discriminative structures. On the reported benchmark, FishViT attains $i\in\{3,4,5\}$9, $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$0, $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$1, $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$2, and $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$3 FPS. In ablation, $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$4 rises from $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$5 for the baseline to $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$6 with STAttention, $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$7 with the multi-level encoder, and $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$8 for the full model (Li et al., 2024).

SG-MFT uses external similarity maps as explicit control signals for multimodal fusion. From Chinese-CLIP text and image features, it forms coarse modality-wise similarity $$\mathbf{M}_i=\mathcal{F}\big(\phi_i(\mathbf{Z})\star \phi_i(\mathbf{X})\big),\quad \mathbf{M}_i\in\mathbb{R}^{WH\times C},$$9 and fine element-wise similarity $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$0 from

$$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$1

The similarity-guided interaction module interpolates cross-attention and self-attention according to

$$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$2

and injects fine-grained cross-modal alignment through

$$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$3

inside a similarity-aware feed-forward block. On the Multimedia Weibo Social Media Dataset, the full SG-MFT reaches $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$4 accuracy, compared with $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$5 for Baseline + CLIP + SIM and $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$6 for Baseline + CLIP + SFM (Zhang et al., 2024).

The commonality between FishViT and SG-MFT is architectural. Neither merely visualizes attention after training. Instead, both alter the transformer computation itself by introducing similarity-dependent gating, shrinkage, or interpolation. In this regime, the similarity map is an operator on attention, not just an interpretive overlay.

5. Graph, geometric, and measure-theoretic extensions

In graph-based similarity learning, Pieceformer defines similarity through graph-level embeddings $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$7 and L2 distances,

$$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$8

Its encoder combines message passing with a Partitioned Graph Transformer using linear attention and METIS-based partitioning. The transformer therefore operates on partition-local node interactions, while the resulting graph embeddings induce a corpus-level similarity matrix $$\mathbf{M}_E^1=\mathrm{Norm}(\mathbf{M}_3'+\mathbf{M}_4'),$$9. The reported partition-size ablation finds the best MAE around $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$0 nodes per partition. Across scales, GIN+PGT achieves the lowest MAE across nearly all settings, with a $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$1 average MAE reduction over GIN across all scales, and the partitioning case study reports up to $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$2 runtime reduction (Yang et al., 18 Jun 2025).

TGO-II shifts from architectural usage to geometric analysis. It computes layer–layer similarity with CKA and SVCCA, token–token similarity with token covariance and a coupling metric, and manifold complexity with TwoNN-ID. For ViT-S/16 on ImageNet-100, mean CKA and mean SVCCA decrease throughout training, intrinsic dimensionality increases before stabilizing, and token covariance retains substantial off-diagonal structure. The paper’s interpretation is that representational specialization and manifold expansion emerge together, while strong token interaction structure persists (Kapil et al., 2 Jul 2026).

At the most abstract level, the measure-theoretic account of transformers defines an in-context map $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$3 and a push-forward measure map

$$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$4

Within this formulation, self-attention appears as a similarity kernel inside $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$5, and the central characterization theorem states that continuous representations of this kind are equivalent to support-preserving maps whose regular part of the Fréchet derivative is uniformly continuous. The same framework further identifies the infinite-depth, mean-field measure-theoretic transformer with a Vlasov flow (Furuya et al., 30 Sep 2025).

These extensions broaden the meaning of the topic. In Pieceformer, the similarity map is induced downstream by transformer-produced embeddings. In TGO-II, similarity maps become observables of representation geometry. In the measure-theoretic setting, similarity is the kernel that defines the transport field of the transformer itself.

6. Design patterns, misconceptions, and limitations

Across the literature, several design patterns recur. One pattern converts task-specific similarity structures into tokens, as in HiFT’s hierarchical cross-correlation maps (Cao et al., 2021). A second computes a scalar similarity score from a pairwise transformer input and then reuses internal structures as explanations, as in CEViT and BERT-based paragraph similarity interpretation (Zhao et al., 2024, Malkiel et al., 2022). A third injects external similarity maps into attention or feed-forward computation, as in FishViT and SG-MFT (Li et al., 2024, Zhang et al., 2024). A fourth uses transformer-produced embeddings to induce a downstream similarity map over a corpus, as in Pieceformer (Yang et al., 18 Jun 2025).

A common misconception is that the topic is exhausted by standard $$\mathbf{M}_E^2=\mathrm{mAtt}(\mathbf{M}_E^1,\mathbf{M}_E^1,\mathbf{M}_3').$$6 attention. The literature is broader. HiFT applies the transformer to correlation maps rather than raw backbone features (Cao et al., 2021). CEViT does not introduce an explicit cross-image attention layer; it fuses the image pair by channel concatenation before patchification (Zhao et al., 2024). The BERT-based text-similarity method uses activation and saliency maps to explain a similarity score after the fact (Malkiel et al., 2022). TGO-II studies similarity maps of representations themselves rather than task-level similarity heads (Kapil et al., 2 Jul 2026). A plausible implication is that “transformer over similarity maps” is best understood as a computational stance toward similarity-structured intermediate objects, not as a single attention formula.

The empirical scope remains uneven. CEViT is reported only on MNIST and is explicitly described as preliminary research (Zhao et al., 2024). SG-MFT depends on similarity estimates supplied by Chinese-CLIP (Zhang et al., 2024). FishViT notes limitations under extreme occlusions, low contrast, and domain shift (Li et al., 2024). Pieceformer identifies edge-feature integration and adaptive partitioning as open directions (Yang et al., 18 Jun 2025). TGO-II studies one architecture, ViT-S/16, on ImageNet-100 (Kapil et al., 2 Jul 2026). The broader field therefore supports the design principle strongly, but its instantiations remain domain-specific.

Taken together, these works establish a coherent technical idea: transformers can operate not only on appearance or token content, but on explicit similarity structures that encode correspondence, alignment, match evidence, or representational geometry. In some systems this produces more efficient multi-level fusion, in others more faithful similarity scoring, more discriminative attention, more interpretable explanations, or a clearer theoretical description of self-attention itself.