---
title: Siamese Dual-Encoder Architectures
url: https://www.emergentmind.com/topics/siamese-dual-encoder-sde-f068776f-2d88-45b7-bd7f-ec5a15173747
type: topic
---

# Siamese Dual-Encoder Architectures

Searching arXiv for recent and foundational papers relevant to Siamese Dual-Encoder formulations and terminology.
First, I’ll look for papers explicitly using “dual encoder” and “Siamese” terminology in retrieval and representation learning.
Siamese Dual-Encoder (SDE) denotes a dual-encoder architecture in which two inputs are processed in parallel by encoder branches that share parameters, so that the resulting representations inhabit a common or comparable feature space. In the retrieval literature, this setting is explicitly contrasted with the Asymmetric Dual Encoder (ADE), whose towers are separately parameterized; in that literature, SDE is also called a Symmetric Dual Encoder, and sometimes a Siamese or Twin dual encoder [2204.07120, 2306.02516]. Across recent arXiv work, however, SDE is better understood as an architectural principle rather than a single task family: the same shared-weight two-branch pattern appears in dense retrieval, long-document matching, remote-sensing change detection, medical imaging, time-series augmentation, protein pretraining, and dual-arm manipulation [2004.12297, 2208.06293, 2301.12068, 2506.13428].

## 1. Terminology and formal definition

In its strict form, an SDE applies the same encoder \(E(\cdot;\theta)\) to both sides of a pair. In change detection, for example, the bi-temporal inputs \(X^{\text{pre}}\) and \(X^{\text{post}}\) are encoded as
\[
F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),
\]
with shared \(\theta\) across both branches, so that corresponding layers operate in aligned feature spaces [2204.09387]. In retrieval-oriented SDEs, the paired outputs are usually compared by a simple shared-space similarity such as cosine similarity,
\[
sim(q_i,a_j) = \frac{q_i}{\|q_i\|}\cdot \frac{a_j}{\|a_j\|},
\]
after independent encoding of query and answer candidates [2204.07120].

This shared-parameter principle is the central inductive bias. It enforces representational comparability across time, modality role, or view, while preserving the operational advantage of separate encoding. In the strongest formulations, the two towers are entirely tied; in weaker formulations, the backbone remains common while small task-specific modules differ across sides. The latter design appears in semi-Siamese bi-encoders, where the main BERT language model is kept common for both query and document while separate lightweight fine-tuning modules introduce a limited amount of asymmetry [2110.14943].

The literature also shows that the term “Siamese” is not always used with identical strictness. StrokeNeXt is described as a Siamese-encoder approach and has two architecturally identical ConvNeXt branches, but the paper is explicit that their parameters are not shared; it therefore fits more precisely under a dual-encoder with Siamese symmetry than a strict tied-weight Siamese network [2602.15087]. This usage indicates that, in current practice, SDE sometimes names a symmetric two-branch computation pattern even when full weight tying is relaxed.

## 2. Recurrent architectural patterns

SDE architectures span a broad design space. Some retain the classical late-interaction pattern of independent encoding followed by a single similarity or scoring head. Others introduce repeated cross-branch interaction at multiple scales, or attach decoders, aligners, or diffusion policies to the paired latent streams.

| Pattern | Representative paper | Characteristic |
|---|---|---|
| Late-interaction retrieval | "Exploring Dual Encoder Architectures for Question Answering" [2204.07120] | Mean-pooled T5 encodings compared in a shared space |
| Hierarchical document matching | "Beyond 512 Tokens" [2004.12297] | Shared long-document towers with sentence-block and document-level Transformers |
| Cross-scale dense prediction | "dual unet" [2208.06293] | Shared-weight dual-stream encoder-decoder with repeated temporal interaction |
| Coarse-to-fine alignment | "Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision" [1904.02643] | Shared multiscale encoders feeding recursive residual aligners |

The simplest SDE pattern remains “encode separately, compare once.” This is the form assumed by standard QA retrieval bi-encoders, by MatchBERT-like document matching systems, and by template-based traffic sign models in which a sample image and a class template are independently mapped into a 256-dimensional feature code and compared with Euclidean distance [2204.07120, 2502.15307]. In these settings, the encoder is the primary object of reuse, and the interaction head is deliberately cheap.

A more interactive pattern appears in dense correspondence and change detection. Dual-UNet is a shared-weight bilateral U-Net network in which encoder features are differenced at every scale, fused back into the branch features, refined by the multiscale differential attention module (MDAM), and then re-compared in the decoder through weighted difference fusion maps [2208.06293]. SEAMLeSS similarly begins with two shared encoders, but then couples the branches through coarse-to-fine geometric warping and residual displacement estimation at each scale rather than through a single terminal similarity [1904.02643]. These examples show that SDE does not imply late interaction.

A third pattern attaches a decoder or generative head to the shared encoder pair. SidAE combines a SimSiam-style tied encoder with a predictor and a decoder that reconstructs the original image from each branch latent [2304.02549]. SiamJEPA replaces the single student encoder of JEPA-style pretraining with two masked student branches that share weights and are regularized by a KL-based cross-view consistency term [2607.04044]. VLM-SFD uses a Siamese observation encoder for two target objects and conditions a diffusion model to predict two object-centric motion flows, which are later assigned to robot arms by a VLM-based controller [2506.13428].

## 3. Objectives, supervision, and embedding geometry

SDE is an architectural pattern rather than a single optimization recipe. The losses used across the literature range from sampled-softmax ranking objectives to reconstruction, contrastive metric learning, diffusion denoising, and latent prediction.

In retrieval, a standard training objective is in-batch sampled softmax over paired query-document examples, with cosine similarity as the scoring function [2204.07120]. SamToNe modifies this objective by adding same-tower negatives to the denominator, for example terms of the form \(e^{\mathtt{sim}(q_i,q_j)/\tau}\), so that a query is pushed away not only from irrelevant documents but also from other queries in the batch [2306.02516]. On large SDE QA retrieval models, this raised average \(P@1/MRR\) from \(44.7/57.5\) to \(46.4/58.7\), and on BEIR it improved SDE-XXL average \(NDCG@10\) from \(46.9\) to \(48.3\) [2306.02516].

In self-supervised image representation learning, SidAE uses a convex combination of reconstruction and Siamese alignment,
\[
L_{\operatorname{sidae}} = w\,L_{\operatorname{dae}} + (1-w)\,L_{\operatorname{si}},
\]
where the Siamese term is SimSiam-style negative cosine similarity and the decoder reconstructs the original image rather than the augmented view [2304.02549]. SiamJEPA uses a JEPA-style latent prediction term over masked regions together with a KL regularizer between branch-conditioned prior and posterior distributions, and reports that stronger Siamese regularization substantially improves early linear-probe accuracy [2607.04044].

In dense prediction, SDE losses are often explicitly imbalance-aware. Dual-UNet uses a batch-balanced contrastive loss with changed and unchanged pixels normalized by their batch frequencies and a margin \(m=2\), rather than ordinary cross-entropy [2208.06293]. In time-series augmentation, DTWSSE trains a Siamese encoder with a metric-regression loss
\[
L_E=\frac{1}{|\mathcal D|}\sum_i \left(\|\mathbf{h}_i^1-\mathbf{h}_i^2\|_2 - y_i\right)^2,
\]
so that Euclidean distance in latent space approximates Dynamic Time Warping (DTW), after which interpolation is performed in latent space and decoded back to time series [2108.09885].

Protein pretraining pushes the idea further away from conventional similarity learning. SiamDiff uses the same sequence-structure encoder on two correlated conformers and optimizes a bidirectional denoising objective that lower-bounds mutual information between diffusion trajectories, rather than using cosine similarity, negatives, or a momentum target encoder [2301.12068]. A plausible implication is that SDE geometry can be induced by shared denoising semantics, not only by explicit embedding-space contrast.

## 4. Retrieval and ranking formulations

The retrieval literature provides the clearest explicit definition of SDE. In question answering, a Siamese Dual Encoder shares the token embedder, transformer encoder, and projection layer across the question and answer towers, and the resulting embeddings are compared in a shared space [2204.07120]. That paper reports that SDE performs significantly better than ADE on QA retrieval tasks, and further finds that sharing only the projection layer in an otherwise asymmetric model can make ADE competitive with, or slightly stronger than, SDE on some settings [2204.07120]. The associated t-SNE analysis indicates that shared projection layers help bring question and answer embeddings into a coinciding parameter space.

This retrieval view extends naturally to long-form document matching. SMITH is a Siamese hierarchical Transformer encoder in which each tower independently encodes a long document through sentence-block-level and document-level Transformer stacks, applies L2 normalization, and computes similarity in the resulting document space [2004.12297]. It increases maximum input text length from 512 to 2048 tokens and reports best \(F1\) values of \(0.9591\) on Wiki65K and \(0.8543\) on AAN104K [2004.12297]. The key point is architectural: SDE scaling to long documents does not require abandoning independent encoding, but it does require a stronger per-document encoder.

Later retrieval work weakens strict symmetry without fully abandoning it. Semi-Siamese bi-encoders keep the pretrained BERT backbone common while learning tiny query-specific and document-specific lightweight fine-tuning modules, amounting to less than \(1\%\) different parameters [2110.14943]. On ClueWeb09b, this limited asymmetry is especially helpful, with semi-Siamese ColBERT outperforming both full fine-tuning and Siamese lightweight baselines [2110.14943]. The broad lesson is that SDE and ADE need not be treated as an all-or-nothing dichotomy.

## 5. Scientific and engineering applications beyond retrieval

Outside retrieval, SDE serves as a general mechanism for comparing structured observations of the same entity across time, condition, or view. In remote sensing change detection, Dual-UNet is a genuine Siamese dual-encoder-decoder architecture with shared weights and recurrent cross-temporal fusion. On the CDD dataset it reports Precision \(97.46\), Recall \(94.31\), \(F1\) \(95.86\), and IoU \(92.06\), outperforming FC-EF, FC-Siam-diff, FC-Siam-conc, and STANet [2208.06293]. The Attentive Dual Stream Siamese U-net for flood detection likewise uses two weight-shared encoders on pre- and post-flood Sentinel-1 SAR inputs and reports IoU \(0.70\) and \(F1\) \(0.83\), a \(6\%\) IoU improvement over a uni-temporal flood detector [2204.09387].

Medical imaging supplies both strict and relaxed variants. The lung-nodule growth model based on a Siamese encoder and spatial-temporal mixer processes paired 3D ROIs from consecutive CT scans, extracts \(F_{L0}\), \(F_{L1}\), and \(F_{G1}\), and combines them through
\[
F_{L1} + LP(Sim(Q_{L1}, K_{G1})\cdot F_{G1} + Sim(Q_{L1}, K_{L0}) \cdot F_{L0}),
\]
with a hierarchical loss that prioritizes growing nodules [2206.03049]. StrokeNeXt, by contrast, uses two ConvNeXt encoders on the same CT image without weight sharing, then fuses the embeddings with a \(k=2\) Conv1D and a bottleneck \(C \rightarrow H \rightarrow C\); it reaches accuracies and \(F1\)-scores of up to \(0.988\) [2602.15087]. Together these papers show that current medical literature treats “Siamese” as a continuum from strict tied encoders to symmetric untied dual encoders.

Other domains expose the same architectural motif under different objectives. DTWSSE uses a Siamese encoder plus decoder so that time-series interpolation can occur in a Euclidean latent space aligned with DTW, rather than directly in input space [2108.09885]. IECES uses a sample branch and a template branch with shared Efficient-CNN-based encoders, contrastive loss, and a final SoftMax classifier; after training, the template branch is discarded to improve runtime, yielding a \(2.9\)M-parameter real-time traffic sign recognizer [2502.15307]. In robotics, VLM-SFD employs a Siamese Flow Diffusion Network that embeds two target objects into a shared latent space and predicts two motion flows for later arm assignment [2506.13428]. In protein modeling, SiamDiff uses two diffusion-corrupted conformers of the same protein and a shared encoder to maximize information useful for denoising both sequence and structure [2301.12068].

## 6. Variants, misconceptions, and limitations

A common misconception is that SDE always means “encode each input independently and compare once at the end.” Dual-UNet and SEAMLeSS are clear counterexamples: both begin with shared encoders, but both introduce repeated cross-branch interaction during refinement, whether by encoder-scale difference maps, attention, and decoder-side fusion, or by multiscale warping and residual displacement prediction [2208.06293, 1904.02643]. In these systems, the Siamese property guarantees feature comparability, but the downstream computation is deeply interactive.

A second misconception is that “Siamese” always implies complete weight sharing. That is true for the strict SDE definition used in QA retrieval, in long-document matching, and in classical change detection [2204.07120, 2004.12297]. It is not true for all contemporary usage. StrokeNeXt explicitly uses untied but symmetric branches, while semi-Siamese bi-encoders share the backbone and introduce only a small amount of query- or document-specific adaptation [2602.15087, 2110.14943]. The literature therefore supports a narrower definition, in which SDE means full parameter tying, and a broader usage, in which Siamese names branch symmetry plus strong representational coupling.

The empirical record also shows practical limitations. Some application papers leave crucial implementation details underspecified, which can complicate faithful reproduction; Dual-UNet is explicit about several modules but leaves parts of the attention and fusion formulas unspecified [2208.06293]. Some loss modifications are data-regime dependent: SamToNe improves SDE retrieval on average, but document-side same-tower negatives can hurt when many training examples share duplicate documents [2306.02516]. A plausible implication is that SDE behavior is governed not only by tower sharing, but by the interaction between sharing pattern, loss geometry, and the structure of paired supervision.

Taken together, the recent literature presents SDE not as a single model class with a fixed scoring rule, but as a reusable architectural prior. Its defining commitment is that paired inputs should first be expressed through a common encoder geometry; what happens after that may be cosine retrieval, multiscale differencing, cross-view denoising, hierarchical decoding, or diffusion-based policy generation.

Source: https://www.emergentmind.com/topics/siamese-dual-encoder-sde-f068776f-2d88-45b7-bd7f-ec5a15173747