Adjusted Siamese Network Overview

• Adjusted Siamese Networks are a family of architectures that preserve shared representation learning while modifying loss functions, training loops, and branch interactions to better align with diverse tasks.
• Adjustments include incorporating modified distance losses, mutual-attention mechanisms, branch asymmetry, and learned decision heads to address challenges like label scarcity and late feature fusion.
• Empirical evaluations across domains such as emotion recognition, remote sensing, tracking, and biometric verification demonstrate improved performance and robustness compared to standard Siamese models.

Searching arXiv for recent and relevant papers on “Adjusted Siamese Network” and the cited variants. Automatic emotion recognition, semi-supervised learning, biometric verification, remote-sensing change detection, visual tracking, self-supervised representation learning, point-cloud completion, image collection, text duplicate detection, and image extrapolation have all produced domain-specific reinterpretations of the Siamese paradigm. Across this literature, an Adjusted Siamese Network denotes not a single canonical architecture but a family of modifications applied to the standard twin-branch, shared-weight design in order to address concrete deficiencies of plain pairwise similarity learning. These adjustments may target the loss, the training loop, the interaction pattern between branches, the role asymmetry of the two streams, the modality-specific preprocessing, or the downstream decision head. In that sense, the term names a design strategy: preserve the Siamese principle of shared representation learning while altering its objective or computation so that it better matches the structure of the task (Feng et al., 2020, Sahito et al., 2021, Zhou et al., 2022).

1. Definition and conceptual scope

A standard Siamese network consists of two identical branches with shared weights that map paired inputs to embeddings whose distance or similarity drives training and inference. The adjusted variants surveyed in the literature preserve some or all of this structure, but depart from the classical formulation in ways that are task-specific.

In speech emotion recognition, the adjustment is a modified distance loss used during transfer-learning fine-tuning, where the network is optimized using the relevant distance between same and different class pairs rather than only ordinary classification loss (Feng et al., 2020). In semi-supervised learning, the adjustment is procedural: the network is trained as a metric learner with triplet loss, then used with nearest-neighbor pseudo-labeling in an iterative self-training loop (Sahito et al., 2021). In remote sensing, one line of work inserts attention gates or mutual-attention between or after encoder stages so that the two branches exchange information earlier than in a vanilla Siamese design (Heidary et al., 2021, Zhou et al., 2022). In object tracking, the adjustment is a Compact Latent Network (CLNet) that predicts lightweight per-sequence parameter deviations from first-frame statistics (Dong et al., 2023).

Other adjusted forms broaden the architectural notion of Siamese processing itself. ASFM-Net is “asymmetrical” because one branch is trained on complete point clouds and then frozen, while the other aligns partial shapes to that latent space (Xia et al., 2021). MultiSiam generalizes the twin-input design to 2 or more input fields, thereby moving from pairwise to group-based duplicate learning (Bhoi et al., 2024). SiENet embeds Siamese feature alignment inside a GAN-based outpainting pipeline, where latent features of covered/generated and ground-truth inputs are brought together to strengthen border extrapolation (Zhang et al., 2020).

This diversity suggests that “adjusted” is best understood as an umbrella term for Siamese architectures modified to overcome one or more of the following standard limitations: dependence on scarce labels, late fusion between branches, inability to exploit unlabeled data, mismatch between verification loss and downstream objective, brittleness under domain shift, or inadequacy of simple embedding distance as a final decision rule.

2. Architectural patterns of adjustment

One recurrent adjustment is to keep the two-branch shared-weight encoder intact while changing how branch outputs are fused or consumed. In urban change detection, the Fully Convolutional Siamese Concatenate Network (FC-Siam-Conc) is extended with attention gate layers in the decoder so that relevant spatial regions are emphasized before feature fusion (Heidary et al., 2021). In MASNet, the change is earlier and more explicit: mutual-attention is inserted after every feature extraction block, allowing the two branches to exchange information during encoding rather than only near the decoder head (Zhou et al., 2022). This is a direct response to the critique that vanilla Siamese change-detection pipelines extract features independently and fuse them too late.

A second pattern is asymmetry within an otherwise Siamese scaffold. SiameseIM uses an online branch and a target branch; the former encodes a masked augmented view and predicts dense representations for the other view, while the latter, updated by EMA, supplies the target dense features (Tao et al., 2022). ASFM-Net likewise breaks symmetry, but in a different way: the complete-shape auto-encoder is trained first and frozen, and the partial-shape branch is optimized to match its latent representation, after which the frozen complete decoder is used to reconstruct a coarse completion (Xia et al., 2021). In both cases, the Siamese template remains recognizable, but the roles of the two branches are no longer equivalent.

A third pattern is augmentation around the Siamese core. PVSNet places the matcher inside a larger two-stage system: an encoder-decoder first learns domain-specific palm-vein transformations, and the Siamese CNN is pretrained as an autoencoder before triplet-based biometric discrimination (Thapar et al., 2018). The paper explicitly characterizes the overall adjustment as domain-specific preprocessing + autoencoder pretraining + triplet training + hard-negative mining + adaptive margin. Similarly, the YOLO-plus-Siamese image collection framework does not use the Siamese model in isolation; YOLOv10 first detects and crops objects, and the Siamese network then performs binary reclassification by similarity to an anchor image (Shin et al., 2024).

A fourth pattern replaces simple distance-based comparison with a learned fusion or decision head. The two-stream vehicle re-identification network processes coarse vehicle-shape patches and license-plate patches in parallel and merges Siamese distance descriptors with a sequence of fully connected layers to reduce false alarms caused by nearly identical vehicle designs or similar license plate strings (Oliveira et al., 2019). This suggests that some adjusted Siamese models are motivated not by representation learning alone but by the insufficiency of raw embedding distance as a decision statistic.

3. Loss functions and optimization strategies

A central axis of adjustment is the loss. In low-label speech emotion recognition, the defining modification is a batch-level relative-distance ratio applied only to the feature extraction layers at the end of a batch during fine-tuning (Feng et al., 2020). Let $\mathcal{X}_s$ be the set of same-class pairs and $\mathcal{X}_d$ the set of different-class pairs in a batch, and let $g_\mathbf{W}(\mathbf{x})$ denote the feature extractor parameterized by $\mathbf{W}$. The objective is written as

$\mathbf{W^*} = \arg\min_{\mathbf{W} \frac{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 + \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 - \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }.$

This differs conceptually from a standard contrastive formulation such as

$L = y\,D^2 + (1-y)\max(0,m-D)^2,$

because it optimizes the relative separation of average same-class and different-class distances rather than enforcing a fixed margin pair by pair (Feng et al., 2020).

Triplet-based objectives constitute another major family. The semi-supervised metric-learning method uses the triplet margin loss

$\mathcal{L} = \max(d(a,p) - d(a,n) + m, 0),$

with Euclidean distance in embedding space and all valid triplets drawn from the labeled set (Sahito et al., 2021). PVSNet employs a triplet Siamese matching network with hard negatives and an adaptive margin. It defines

$J_{i} = L_{2}^{2}(a,i)$

and

$D(a,p,hn) = \frac{1}{2}{max(0,M+J_{p}-J_{hn})},$

where $M$ is the margin, $\mathcal{X}_d$0 is a positive image of the same subject, and $\mathcal{X}_d$1 is a hard negative from a different subject (Thapar et al., 2018). The margin is dynamically increased from $\mathcal{X}_d$2 to $\mathcal{X}_d$3, and the negatives are mined online before forming each batch.

Contrastive learning remains important in adjusted Siamese systems used for reclassification or verification. The image collection framework defines the Euclidean distance

$\mathcal{X}_d$4

between a candidate image and an anchor image and trains with

$\mathcal{X}_d$5

where $\mathcal{X}_d$6 for similar pairs and $\mathcal{X}_d$7 for dissimilar pairs (Shin et al., 2024).

Other adjusted networks combine Siamese alignment with additional objectives. HSSNet minimizes a logistic loss over spatial response maps for similarity verification in thermal infrared tracking (Li et al., 2017). SiENet supplements adversarial, distance, perceptual, and style losses with a Siamese alignment term

$\mathcal{X}_d$8

to bring encoder features of covered/generated input closer to those of ground-truth input (Zhang et al., 2020). SiameseIM uses a dense UniGrad loss over token representations so that a masked augmented view predicts the dense representation of another augmented view (Tao et al., 2022). These examples show that adjustment frequently means embedding the Siamese principle inside a broader multi-loss optimization program rather than treating pairwise distance as the sole supervisory signal.

4. Training-loop and data-regime adaptations

Adjusted Siamese networks are often responses to data scarcity, label scarcity, or domain mismatch. In speech emotion recognition, the target dataset is small, and the system therefore uses source data for pre-training and a small labeled subset of the target domain for fine-tuning (Feng et al., 2020). The main fine-tuning setting selects one sample per emotion from two randomly chosen speakers, giving 8 labeled target samples total, while also varying the number of adaptation speakers from 2, 4, ..., 18, 20 speakers and the number of frozen layers during fine-tuning (Feng et al., 2020). The rationale is explicit: pairwise modeling can exploit “same emotion vs. different emotion” relations more efficiently than single-example classification in few-shot conditions.

In semi-supervised learning, the adjustment is not to architecture but to the lifecycle of training. The network is first trained on labeled data only, embeddings are computed for labeled and unlabeled sets, labels for unlabeled data are predicted with nearest neighbors, and a fixed percentage $\mathcal{X}_d$9 of the most confident pseudo-labeled points is added to the labeled pool. This is repeated for 25 meta-iterations after 200 epochs of Siamese training, with Adam optimizer, triplet loss margin $g_\mathbf{W}(\mathbf{x})$0, no data augmentation, and final test accuracy computed with $g_\mathbf{W}(\mathbf{x})$1 nearest neighbor (Sahito et al., 2021). The method can optionally refine pseudo-labels by local learning with global consistency (LLGC), though the reported gains are limited on some datasets (Sahito et al., 2021).

Biometric settings motivate yet another staged training strategy. PVSNet first pretrains the encoder of the Siamese feature extractor as an autoencoder on the multi-channel feature image, then discards the decoder and uses the encoder weights to initialize the matching network (Thapar et al., 2018). The convolutional layers are initially frozen while the fully connected layers are trained; only later are all layers fine-tuned, because loss from an untrained fully connected layer could distort the generative features learned during pretraining (Thapar et al., 2018).

Tracking introduces sequence-specific rather than dataset-level adaptation. The CLNet framework for adjusted Siamese tracking treats the first frame as a support set containing decisive samples from the current scene. It computes a statistics-based compact latent feature from positive and negative features,

$g_\mathbf{W}(\mathbf{x})$2

for $g_\mathbf{W}(\mathbf{x})$3, concatenates these into $g_\mathbf{W}(\mathbf{x})$4, and predicts lightweight parameter deviations for the last layer of the tracker (Dong et al., 2023). Adaptation therefore becomes a fast statistical adjustment rather than slow online gradient descent.

These cases show that “adjusted” often refers as much to training protocol and data usage as to network topology.

5. Cross-branch communication, attention, and asymmetry

A major criticism of vanilla Siamese encoders in change detection is that the two branches do not interact until the decoder. The response has been the insertion of explicit cross-branch communication. In the urban change-detection model built on FC-Siam-Conc, Gaussian Attention (GA) is introduced as a preprocessing step, but because GA parameters cannot be adjusted during training, a learnable attention gate layer is added in the decoder to play the role of GA with tunable parameters (Heidary et al., 2021). The result is a Siamese encoder-decoder in which skip features are spatially reweighted before propagation.

MASNet pushes this logic further. Its mutual-attention plug-in exchanges information between feature maps $g_\mathbf{W}(\mathbf{x})$5 and $g_\mathbf{W}(\mathbf{x})$6 after every feature extraction block:

$g_\mathbf{W}(\mathbf{x})$7

$g_\mathbf{W}(\mathbf{x})$8

with branch updates

$g_\mathbf{W}(\mathbf{x})$9

and

$\mathbf{W}$0

The authors emphasize that this is not self-attention but cross-image conditioning, and they use individual-level mutual-attention to limit parameters and FLOPs (Zhou et al., 2022). This suggests that one widely accepted meaning of “adjusted Siamese network” in remote sensing is a Siamese backbone whose branches are no longer informationally independent.

Asymmetry serves a related but distinct role. In ASFM-Net, the complete-point-cloud auto-encoder learns a latent manifold of valid full shapes, and the partial branch is forced to align to it with the Euclidean feature matching loss

$\mathbf{W}$1

Only the partial branch is needed at inference time, while the frozen decoder of the complete branch reconstructs a coarse completion (Xia et al., 2021). SiameseIM also replaces symmetry with an online-target distinction tied to dense view-to-view prediction and relative positional correspondence (Tao et al., 2022). These systems indicate that adjusted Siamese designs increasingly treat the two branches as functionally different operators within a shared latent space rather than mere clones used for distance computation.

6. Representative applications and empirical behavior

The empirical profile of adjusted Siamese networks varies substantially across domains, but the reported results consistently support the view that the adjustments matter most when the vanilla Siamese setup is mismatched to task structure.

In speech emotion recognition transfer, out-of-domain training on RAVDESS performs poorly, with 32.8% UAR when the source is eNTERFACE’05 and 29.3% UAR when the source is CREMA-D. Standard Siamese fine-tuning improves to 32.9% with eNTERFACE’05 and 37.8% with CREMA-D, while adding the modified distance loss raises performance to 39.9% and 43.3–43.4%, respectively. The in-domain leave-one-subject-out upper bound on RAVDESS is 50.0% UAR (Feng et al., 2020). The paper also reports that the choice of source data has more effect on final UAR than whether zero, one, or two layers are frozen, and that roughly 3–5 speakers in target fine-tuning data give the best trade-off between performance and data efficiency (Feng et al., 2020).

In semi-supervised learning, Siamese self-training substantially improves over supervised-only baselines under scarce labels. On MNIST with 100 labeled examples, error drops from $\mathbf{W}$2 to $\mathbf{W}$3; on Fashion-MNIST with 100 labeled examples, from $\mathbf{W}$4 to $\mathbf{W}$5; on SVHN with 1000 labeled examples, from $\mathbf{W}$6 to $\mathbf{W}$7; and on CIFAR-10 with 4000 labeled examples, from $\mathbf{W}$8 to $\mathbf{W}$9 (Sahito et al., 2021). LLGC sometimes helps slightly, but often $\mathbf{W^*} = \arg\min_{\mathbf{W} \frac{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 + \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }{ \frac{1}{\|\mathcal{X}_d\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_d}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 - \frac{1}{\|\mathcal{X}_s\|_0}\sum_{\mathbf{x},\mathbf{x}'\in\mathcal{X}_s}\|g(\mathbf{x})-g(\mathbf{x}')\|_2 }.$0-NN-based self-training already captures most of the gain (Sahito et al., 2021).

In biometric authentication, PVSNet reports intra-dataset EERs of 3.71% on CASIA, 0.93% on IITI, and 0.66% on PolyU, outperforming FaceNet on all three. Inter-dataset performance drops, but PVSNet still beats FaceNet, for example 9.37% EER vs 12.72% for Train IITI / Test CASIA (Thapar et al., 2018). This supports the claim that adaptive-margin triplet learning and hard-negative mining improve discrimination under limited biometric data.

In remote-sensing change detection, the attention-based FC-Siam variants improve over their non-attention counterparts on OSCD and RIVER-CD. On OSCD, FC-Siam-diff-Att achieves F1 = 69.39, Precision = 67.47, Recall = 71.86, and Accuracy = 93.33, while FC-Siam-conc-Att reaches F1 = 67.96, Recall = 73.07, and Accuracy = 92.13 (Heidary et al., 2021). On RIVER-CD, FC-Siam-diff-Att-GA achieves F1 = 88.46, Precision = 90.56, and Recall = 86.6 (Heidary et al., 2021). MASNet improves both CNN- and ViT-based backbones: on SECOND, HRNet-OCR rises from 53.70 mIoU to 55.59 mIoU, and SegFormer from 51.73 mIoU to 55.27 mIoU; on LEVIR-CD, HRNet-OCR improves from 85.45 IoU to 85.79 IoU, and SegFormer from 84.32 IoU to 85.00 IoU (Zhou et al., 2022).

In tracking, the adjusted trackers based on CLNet preserve real-time speed—38.1 FPS for CLNet*-RPN, 104.9 FPS for CLNet*-FC, and 43.6 FPS for CLNet*-BAN—while improving precision and AUC on NfS30, DTB70, LaSOT, GOT10k, and VOT benchmarks (Dong et al., 2023). For example, CLNet*-RPN improves SiamRPN++ by +10.6% precision and +10.2% AUC on NfS30 (Dong et al., 2023).

In self-supervised vision representation learning, SiameseIM is reported to surpass both MoCo-v3 and MAE on ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation, with the improvement being more significant in few-shot, long-tail, and robustness-concerned scenarios (Tao et al., 2022). In point-cloud completion, ASFM-Net achieves 1st place on the Completion3D leaderboard, with average CD of 6.68 and about 17.7% improvement over the second-best method VRC-Net on that benchmark, and average CD of 12.09 on PCN, about 6.1% improvement over the second-best method RFA (Xia et al., 2021).

These results suggest that the practical value of adjusted Siamese networks is strongest in tasks where pairwise or view-to-view relational structure is essential, but naive symmetry, late fusion, or fixed losses are inadequate.

7. Interpretation, limitations, and relation to neighboring paradigms

Several misconceptions recur in discussions of adjusted Siamese networks. One is that “adjusted” necessarily implies a new network topology. The literature does not support such a narrow reading. Some papers change the architecture substantially, as in MultiSiam’s multi-input generalization or ASFM-Net’s branch asymmetry (Bhoi et al., 2024, Xia et al., 2021); others keep the architecture almost intact and instead modify the loss, the training loop, or the adaptation strategy, as in speech emotion transfer and semi-supervised self-training (Feng et al., 2020, Sahito et al., 2021). The notion is therefore methodological rather than taxonomic.

A second misconception is that all adjusted Siamese networks remain pure metric learners. In fact, some variants preserve metric learning at the core but add learned decision layers, such as the fully connected fusion head for vehicle re-identification (Oliveira et al., 2019). Others use the Siamese principle in service of generative modeling or self-supervised dense prediction rather than pair verification, as in SiENet and SiameseIM (Zhang et al., 2020, Tao et al., 2022). A plausible implication is that the Siamese idea is increasingly treated as a representational prior—shared encoding under structured relational supervision—rather than as a commitment to a particular inference rule.

A third misconception is that branch symmetry is essential. Multiple papers explicitly relax symmetry. ASFM-Net freezes the complete-shape branch and uses it as a source of latent priors (Xia et al., 2021). SiameseIM uses an online branch and a momentum target branch (Tao et al., 2022). CLNet adjusts only the last layer(s) of the base tracker using sequence-specific latent statistics (Dong et al., 2023). These cases show that asymmetry is often the mechanism that allows a Siamese framework to encode prior knowledge, temporal adaptation, or teacher–student structure.

Limitations remain. In speech emotion recognition, the modified Siamese approach still does not reach the fully in-domain upper bound (Feng et al., 2020). In semi-supervised learning, gains on CIFAR-10 are limited, likely because the simple convolutional embedding is not expressive enough (Sahito et al., 2021). In remote-sensing attention models, recall can be sensitive when attention suppresses weak but real changes (Heidary et al., 2021). In tracking, heavier online-learning trackers or stronger transformer-based methods can surpass CLNet in raw accuracy, albeit often at lower speed (Dong et al., 2023). These observations indicate that adjustment improves task fit but does not abolish the usual trade-offs between expressivity, efficiency, and robustness.

Taken together, the literature portrays the Adjusted Siamese Network as a flexible research motif rather than a single named model: a Siamese or Siamese-derived system altered so that its inductive bias, optimization target, and interaction structure are better aligned with the relational demands of a specific problem. Under that interpretation, the term unifies a broad set of recent developments in transfer learning, metric learning, self-supervision, change detection, tracking, completion, and multimodal verification (Feng et al., 2020, Zhou et al., 2022, Tao et al., 2022).