---
title: Adjusted Siamese Network Overview
url: https://www.emergentmind.com/topics/adjusted-siamese-network
type: topic
---

# Adjusted Siamese Network Overview

Searching arXiv for recent and relevant papers on “Adjusted Siamese Network” and the cited variants.
Automatic emotion recognition, semi-supervised learning, biometric verification, remote-sensing change detection, visual tracking, self-supervised representation learning, point-cloud completion, image collection, text duplicate detection, and image extrapolation have all produced domain-specific reinterpretations of the Siamese paradigm. Across this literature, an **Adjusted Siamese Network** denotes not a single canonical architecture but a family of modifications applied to the standard twin-branch, shared-weight design in order to address concrete deficiencies of plain pairwise similarity learning. These adjustments may target the loss, the training loop, the interaction pattern between branches, the role asymmetry of the two streams, the modality-specific preprocessing, or the downstream decision head. In that sense, the term names a design strategy: preserve the Siamese principle of shared representation learning while altering its objective or computation so that it better matches the structure of the task [2006.03001], [2109.00794], [2206.02331].

## 1. Definition and conceptual scope

A standard Siamese network consists of two identical branches with shared weights that map paired inputs to embeddings whose distance or similarity drives training and inference. The adjusted variants surveyed in the literature preserve some or all of this structure, but depart from the classical formulation in ways that are task-specific.

In speech emotion recognition, the adjustment is a **modified distance loss** used during transfer-learning fine-tuning, where the network is optimized using the relevant distance between same and different class pairs rather than only ordinary classification loss [2006.03001]. In semi-supervised learning, the adjustment is procedural: the network is trained as a **metric learner** with triplet loss, then used with nearest-neighbor pseudo-labeling in an iterative self-training loop [2109.00794]. In remote sensing, one line of work inserts **attention gates** or **mutual-attention** between or after encoder stages so that the two branches exchange information earlier than in a vanilla Siamese design [2102.00501], [2206.02331]. In object tracking, the adjustment is a **Compact Latent Network (CLNet)** that predicts lightweight per-sequence parameter deviations from first-frame statistics [2302.00930].

Other adjusted forms broaden the architectural notion of Siamese processing itself. ASFM-Net is “asymmetrical” because one branch is trained on complete point clouds and then frozen, while the other aligns partial shapes to that latent space [2104.09587]. MultiSiam generalizes the twin-input design to **2 or more input fields**, thereby moving from pairwise to group-based duplicate learning [2401.06783]. SiENet embeds Siamese feature alignment inside a GAN-based outpainting pipeline, where latent features of covered/generated and ground-truth inputs are brought together to strengthen border extrapolation [2007.03851].

This diversity suggests that “adjusted” is best understood as an umbrella term for Siamese architectures modified to overcome one or more of the following standard limitations: dependence on scarce labels, late fusion between branches, inability to exploit unlabeled data, mismatch between verification loss and downstream objective, brittleness under domain shift, or inadequacy of simple embedding distance as a final decision rule.

## 2. Architectural patterns of adjustment

One recurrent adjustment is to keep the two-branch shared-weight encoder intact while changing how branch outputs are fused or consumed. In urban change detection, the **Fully Convolutional Siamese Concatenate Network (FC-Siam-Conc)** is extended with **attention gate layers** in the decoder so that relevant spatial regions are emphasized before feature fusion [2102.00501]. In MASNet, the change is earlier and more explicit: **mutual-attention** is inserted after every feature extraction block, allowing the two branches to exchange information during encoding rather than only near the decoder head [2206.02331]. This is a direct response to the critique that vanilla Siamese change-detection pipelines extract features independently and fuse them too late.

A second pattern is **asymmetry within an otherwise Siamese scaffold**. SiameseIM uses an online branch and a target branch; the former encodes a masked augmented view and predicts dense representations for the other view, while the latter, updated by EMA, supplies the target dense features [2206.01204]. ASFM-Net likewise breaks symmetry, but in a different way: the complete-shape auto-encoder is trained first and frozen, and the partial-shape branch is optimized to match its latent representation, after which the frozen complete decoder is used to reconstruct a coarse completion [2104.09587]. In both cases, the Siamese template remains recognizable, but the roles of the two branches are no longer equivalent.

A third pattern is **augmentation around the Siamese core**. PVSNet places the matcher inside a larger two-stage system: an encoder-decoder first learns domain-specific palm-vein transformations, and the Siamese CNN is pretrained as an autoencoder before triplet-based biometric discrimination [1812.06271]. The paper explicitly characterizes the overall adjustment as **domain-specific preprocessing + autoencoder pretraining + triplet training + hard-negative mining + adaptive margin**. Similarly, the YOLO-plus-Siamese image collection framework does not use the Siamese model in isolation; YOLOv10 first detects and crops objects, and the Siamese network then performs binary reclassification by similarity to an anchor image [2410.12561].

A fourth pattern replaces simple distance-based comparison with a **learned fusion or decision head**. The two-stream vehicle re-identification network processes coarse vehicle-shape patches and license-plate patches in parallel and merges Siamese distance descriptors with a sequence of fully connected layers to reduce false alarms caused by nearly identical vehicle designs or similar license plate strings [1902.01496]. This suggests that some adjusted Siamese models are motivated not by representation learning alone but by the insufficiency of raw embedding distance as a decision statistic.

## 3. Loss functions and optimization strategies

A central axis of adjustment is the loss. In low-label speech emotion recognition, the defining modification is a batch-level relative-distance ratio applied only to the feature extraction layers at the end of a batch during fine-tuning [2006.03001]. Let $\mathcal{X}_s$ be the set of same-class pairs and $\mathcal{X}_d$ the set of different-class pairs in a batch, and let $g_\mathbf{W}(\mathbf{x})$ denote the feature extractor parameterized by $\mathbf{W}$. The objective is written as
$$
\mathbf{W^*} = \arg\min_{\mathbf{W} \frac{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 + \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 - \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }.
$$
This differs conceptually from a standard contrastive formulation such as
$$
L = y\,D^2 + (1-y)\max(0,m-D)^2,
$$
because it optimizes the **relative separation** of average same-class and different-class distances rather than enforcing a fixed margin pair by pair [2006.03001].

Triplet-based objectives constitute another major family. The semi-supervised metric-learning method uses the triplet margin loss
$$
\mathcal{L} = \max(d(a,p) - d(a,n) + m, 0),
$$
with Euclidean distance in embedding space and all valid triplets drawn from the labeled set [2109.00794]. PVSNet employs a triplet Siamese matching network with hard negatives and an adaptive margin. It defines
$$
J_{i} = L_{2}^{2}(a,i)
$$
and
$$
D(a,p,hn) = \frac{1}{2}{max(0,M+J_{p}-J_{hn})},
$$
where $M$ is the margin, $p$ is a positive image of the same subject, and $hn$ is a hard negative from a different subject [1812.06271]. The margin is dynamically increased from $0.2$ to $0.5$, and the negatives are mined online before forming each batch.

Contrastive learning remains important in adjusted Siamese systems used for reclassification or verification. The image collection framework defines the Euclidean distance
$$
D(I,A) = \lVert f(I) - f(A)\rVert_2
$$
between a candidate image and an anchor image and trains with
$$
\frac{1}{2}\Big(y \cdot D(I,A)^2 + (1-y)\cdot (m - D(I,A))^2\Big),
$$
where $y=1$ for similar pairs and $y=0$ for dissimilar pairs [2410.12561].

Other adjusted networks combine Siamese alignment with additional objectives. HSSNet minimizes a logistic loss over spatial response maps for similarity verification in thermal infrared tracking [1711.09539]. SiENet supplements adversarial, distance, perceptual, and style losses with a Siamese alignment term
$$
L_{siamese} = \|F' - F^{gt}\|_2
$$
to bring encoder features of covered/generated input closer to those of ground-truth input [2007.03851]. SiameseIM uses a dense UniGrad loss over token representations so that a masked augmented view predicts the dense representation of another augmented view [2206.01204]. These examples show that adjustment frequently means embedding the Siamese principle inside a broader multi-loss optimization program rather than treating pairwise distance as the sole supervisory signal.

## 4. Training-loop and data-regime adaptations

Adjusted Siamese networks are often responses to data scarcity, label scarcity, or domain mismatch. In speech emotion recognition, the target dataset is small, and the system therefore uses source data for **pre-training** and a small labeled subset of the target domain for **fine-tuning** [2006.03001]. The main fine-tuning setting selects **one sample per emotion from two randomly chosen speakers**, giving **8 labeled target samples total**, while also varying the number of adaptation speakers from **2, 4, ..., 18, 20 speakers** and the number of frozen layers during fine-tuning [2006.03001]. The rationale is explicit: pairwise modeling can exploit “same emotion vs. different emotion” relations more efficiently than single-example classification in few-shot conditions.

In semi-supervised learning, the adjustment is not to architecture but to the lifecycle of training. The network is first trained on labeled data only, embeddings are computed for labeled and unlabeled sets, labels for unlabeled data are predicted with **nearest neighbors**, and a fixed percentage $p$ of the most confident pseudo-labeled points is added to the labeled pool. This is repeated for **25 meta-iterations** after **200 epochs** of Siamese training, with **Adam optimizer**, **triplet loss margin $m=0.3$**, **no data augmentation**, and final test accuracy computed with **$k=1$ nearest neighbor** [2109.00794]. The method can optionally refine pseudo-labels by **local learning with global consistency (LLGC)**, though the reported gains are limited on some datasets [2109.00794].

Biometric settings motivate yet another staged training strategy. PVSNet first pretrains the encoder of the Siamese feature extractor as an autoencoder on the **multi-channel feature image**, then discards the decoder and uses the encoder weights to initialize the matching network [1812.06271]. The convolutional layers are initially frozen while the fully connected layers are trained; only later are all layers fine-tuned, because loss from an untrained fully connected layer could distort the generative features learned during pretraining [1812.06271].

Tracking introduces sequence-specific rather than dataset-level adaptation. The CLNet framework for adjusted Siamese tracking treats the first frame as a support set containing decisive samples from the current scene. It computes a statistics-based compact latent feature from positive and negative features,
$$
\mu^\rho = \frac{1}{n^\rho}\sum_{i=1}^{n^\rho}\bar{\mathbf{m}_i^\rho,\qquad
\sigma^\rho = \sqrt{\frac{1}{n^\rho}\sum_{i=1}^{n^\rho}(\bar{\mathbf{m}_i^\rho-\mu^\rho)^2}},
$$
for $\rho\in\{+,-\}$, concatenates these into $\mathbf{c}$, and predicts lightweight parameter deviations for the last layer of the tracker [2302.00930]. Adaptation therefore becomes a fast statistical adjustment rather than slow online gradient descent.

These cases show that “adjusted” often refers as much to **training protocol** and **data usage** as to network topology.

## 5. Cross-branch communication, attention, and asymmetry

A major criticism of vanilla Siamese encoders in change detection is that the two branches do not interact until the decoder. The response has been the insertion of explicit cross-branch communication. In the urban change-detection model built on FC-Siam-Conc, **Gaussian Attention (GA)** is introduced as a preprocessing step, but because GA parameters cannot be adjusted during training, a learnable **attention gate layer** is added in the decoder to play the role of GA with tunable parameters [2102.00501]. The result is a Siamese encoder-decoder in which skip features are spatially reweighted before propagation.

MASNet pushes this logic further. Its mutual-attention plug-in exchanges information between feature maps $x_1$ and $x_2$ after every feature extraction block:
$$
q_1, k_1, v_1 = W_q[x_1], W_k[x_1], W_v[x_1]
$$
$$
q_2, k_2, v_2 = W_q[x_2], W_k[x_2], W_v[x_2]
$$
with branch updates
$$
x_1 = x_1 + \text{softmax}\!\left(\frac{q_2 \cdot k_1^T}{\sqrt{d}}\right) v_1
$$
and
$$
x_2 = x_2 + \text{softmax}\!\left(\frac{q_1 \cdot k_2^T}{\sqrt{d}}\right) v_2.
$$
The authors emphasize that this is not self-attention but cross-image conditioning, and they use **individual-level mutual-attention** to limit parameters and FLOPs [2206.02331]. This suggests that one widely accepted meaning of “adjusted Siamese network” in remote sensing is a Siamese backbone whose branches are no longer informationally independent.

Asymmetry serves a related but distinct role. In ASFM-Net, the complete-point-cloud auto-encoder learns a latent manifold of valid full shapes, and the partial branch is forced to align to it with the Euclidean feature matching loss
$$
\mathcal {L}_{feat}(X,Y) = \sum\limits_{i = 0}^n \left\| F_{p_i} - F_{c_i} \right\|_2.
$$
Only the partial branch is needed at inference time, while the frozen decoder of the complete branch reconstructs a coarse completion [2104.09587]. SiameseIM also replaces symmetry with an online-target distinction tied to dense view-to-view prediction and relative positional correspondence [2206.01204]. These systems indicate that adjusted Siamese designs increasingly treat the two branches as functionally different operators within a shared latent space rather than mere clones used for distance computation.

## 6. Representative applications and empirical behavior

The empirical profile of adjusted Siamese networks varies substantially across domains, but the reported results consistently support the view that the adjustments matter most when the vanilla Siamese setup is mismatched to task structure.

In speech emotion recognition transfer, out-of-domain training on RAVDESS performs poorly, with **32.8% UAR** when the source is eNTERFACE’05 and **29.3% UAR** when the source is CREMA-D. Standard Siamese fine-tuning improves to **32.9%** with eNTERFACE’05 and **37.8%** with CREMA-D, while adding the modified distance loss raises performance to **39.9%** and **43.3–43.4%**, respectively. The in-domain leave-one-subject-out upper bound on RAVDESS is **50.0% UAR** [2006.03001]. The paper also reports that the choice of source data has more effect on final UAR than whether zero, one, or two layers are frozen, and that roughly **3–5 speakers** in target fine-tuning data give the best trade-off between performance and data efficiency [2006.03001].

In semi-supervised learning, Siamese self-training substantially improves over supervised-only baselines under scarce labels. On MNIST with **100 labeled examples**, error drops from **$9.73 \pm 0.74\%$** to **$3.24 \pm 0.32\%$**; on Fashion-MNIST with **100 labeled examples**, from **$26.72 \pm 1.23\%$** to **$23.33 \pm 0.43\%$**; on SVHN with **1000 labeled examples**, from **$30.33 \pm 1.55\%$** to **$20.09 \pm 3.22\%$**; and on CIFAR-10 with **4000 labeled examples**, from **$40.87 \pm 0.56\%$** to **$36.56 \pm 0.74\%$** [2109.00794]. LLGC sometimes helps slightly, but often $k$-NN-based self-training already captures most of the gain [2109.00794].

In biometric authentication, PVSNet reports intra-dataset EERs of **3.71%** on CASIA, **0.93%** on IITI, and **0.66%** on PolyU, outperforming FaceNet on all three. Inter-dataset performance drops, but PVSNet still beats FaceNet, for example **9.37% EER vs 12.72%** for Train IITI / Test CASIA [1812.06271]. This supports the claim that adaptive-margin triplet learning and hard-negative mining improve discrimination under limited biometric data.

In remote-sensing change detection, the attention-based FC-Siam variants improve over their non-attention counterparts on OSCD and RIVER-CD. On OSCD, **FC-Siam-diff-Att** achieves **F1 = 69.39**, **Precision = 67.47**, **Recall = 71.86**, and **Accuracy = 93.33**, while **FC-Siam-conc-Att** reaches **F1 = 67.96**, **Recall = 73.07**, and **Accuracy = 92.13** [2102.00501]. On RIVER-CD, **FC-Siam-diff-Att-GA** achieves **F1 = 88.46**, **Precision = 90.56**, and **Recall = 86.6** [2102.00501]. MASNet improves both CNN- and ViT-based backbones: on SECOND, HRNet-OCR rises from **53.70 mIoU** to **55.59 mIoU**, and SegFormer from **51.73 mIoU** to **55.27 mIoU**; on LEVIR-CD, HRNet-OCR improves from **85.45 IoU** to **85.79 IoU**, and SegFormer from **84.32 IoU** to **85.00 IoU** [2206.02331].

In tracking, the adjusted trackers based on CLNet preserve real-time speed—**38.1 FPS** for CLNet*-RPN, **104.9 FPS** for CLNet*-FC, and **43.6 FPS** for CLNet*-BAN—while improving precision and AUC on NfS30, DTB70, LaSOT, GOT10k, and VOT benchmarks [2302.00930]. For example, CLNet*-RPN improves SiamRPN++ by **+10.6% precision** and **+10.2% AUC** on NfS30 [2302.00930].

In self-supervised vision representation learning, SiameseIM is reported to surpass both MoCo-v3 and MAE on ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation, with the improvement being more significant in few-shot, long-tail, and robustness-concerned scenarios [2206.01204]. In point-cloud completion, ASFM-Net achieves **1st place** on the Completion3D leaderboard, with **average CD of 6.68** and about **17.7% improvement** over the second-best method VRC-Net on that benchmark, and **average CD of 12.09** on PCN, about **6.1% improvement** over the second-best method RFA [2104.09587].

These results suggest that the practical value of adjusted Siamese networks is strongest in tasks where pairwise or view-to-view relational structure is essential, but naive symmetry, late fusion, or fixed losses are inadequate.

## 7. Interpretation, limitations, and relation to neighboring paradigms

Several misconceptions recur in discussions of adjusted Siamese networks. One is that “adjusted” necessarily implies a new network topology. The literature does not support such a narrow reading. Some papers change the architecture substantially, as in MultiSiam’s multi-input generalization or ASFM-Net’s branch asymmetry [2401.06783], [2104.09587]; others keep the architecture almost intact and instead modify the loss, the training loop, or the adaptation strategy, as in speech emotion transfer and semi-supervised self-training [2006.03001], [2109.00794]. The notion is therefore methodological rather than taxonomic.

A second misconception is that all adjusted Siamese networks remain pure metric learners. In fact, some variants preserve metric learning at the core but add learned decision layers, such as the fully connected fusion head for vehicle re-identification [1902.01496]. Others use the Siamese principle in service of generative modeling or self-supervised dense prediction rather than pair verification, as in SiENet and SiameseIM [2007.03851], [2206.01204]. A plausible implication is that the Siamese idea is increasingly treated as a representational prior—shared encoding under structured relational supervision—rather than as a commitment to a particular inference rule.

A third misconception is that branch symmetry is essential. Multiple papers explicitly relax symmetry. ASFM-Net freezes the complete-shape branch and uses it as a source of latent priors [2104.09587]. SiameseIM uses an online branch and a momentum target branch [2206.01204]. CLNet adjusts only the last layer(s) of the base tracker using sequence-specific latent statistics [2302.00930]. These cases show that asymmetry is often the mechanism that allows a Siamese framework to encode prior knowledge, temporal adaptation, or teacher–student structure.

Limitations remain. In speech emotion recognition, the modified Siamese approach still does not reach the fully in-domain upper bound [2006.03001]. In semi-supervised learning, gains on CIFAR-10 are limited, likely because the simple convolutional embedding is not expressive enough [2109.00794]. In remote-sensing attention models, recall can be sensitive when attention suppresses weak but real changes [2102.00501]. In tracking, heavier online-learning trackers or stronger transformer-based methods can surpass CLNet in raw accuracy, albeit often at lower speed [2302.00930]. These observations indicate that adjustment improves task fit but does not abolish the usual trade-offs between expressivity, efficiency, and robustness.

Taken together, the literature portrays the Adjusted Siamese Network as a flexible research motif rather than a single named model: a Siamese or Siamese-derived system altered so that its inductive bias, optimization target, and interaction structure are better aligned with the relational demands of a specific problem. Under that interpretation, the term unifies a broad set of recent developments in transfer learning, metric learning, self-supervision, change detection, tracking, completion, and multimodal verification [2006.03001], [2206.02331], [2206.01204].

Source: https://www.emergentmind.com/topics/adjusted-siamese-network