Siamese Dual-Encoder Architectures

• Siamese Dual-Encoder (SDE) is a dual-branch architecture that uses a common encoder to process paired inputs into a unified feature space.
• It is applied in diverse fields such as dense retrieval, long-document matching, medical imaging, and remote sensing, demonstrating its flexibility.
• Recent work shows SDE variants achieve improved performance through strict parameter sharing or controlled cross-branch interactions to suit specific tasks.

Searching arXiv for recent and foundational papers relevant to Siamese Dual-Encoder formulations and terminology. First, I’ll look for papers explicitly using “dual encoder” and “Siamese” terminology in retrieval and representation learning. Siamese Dual-Encoder (SDE) denotes a dual-encoder architecture in which two inputs are processed in parallel by encoder branches that share parameters, so that the resulting representations inhabit a common or comparable feature space. In the retrieval literature, this setting is explicitly contrasted with the Asymmetric Dual Encoder (ADE), whose towers are separately parameterized; in that literature, SDE is also called a Symmetric Dual Encoder, and sometimes a Siamese or Twin dual encoder (Dong et al., 2022, Moiseev et al., 2023). Across recent arXiv work, however, SDE is better understood as an architectural principle rather than a single task family: the same shared-weight two-branch pattern appears in dense retrieval, long-document matching, remote-sensing change detection, medical imaging, time-series augmentation, protein pretraining, and dual-arm manipulation (Yang et al., 2020, Jiang et al., 2022, Zhang et al., 2023, Chen et al., 16 Jun 2025).

1. Terminology and formal definition

In its strict form, an SDE applies the same encoder $E(\cdot;\theta)$ to both sides of a pair. In change detection, for example, the bi-temporal inputs $X^{\text{pre}}$ and $X^{\text{post}}$ are encoded as

$$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$

with shared $\theta$ across both branches, so that corresponding layers operate in aligned feature spaces (Yadav et al., 2022). In retrieval-oriented SDEs, the paired outputs are usually compared by a simple shared-space similarity such as cosine similarity,

$$sim(q_i,a_j) = \frac{q_i}{\|q_i\|}\cdot \frac{a_j}{\|a_j\|},$$

after independent encoding of query and answer candidates (Dong et al., 2022).

This shared-parameter principle is the central inductive bias. It enforces representational comparability across time, modality role, or view, while preserving the operational advantage of separate encoding. In the strongest formulations, the two towers are entirely tied; in weaker formulations, the backbone remains common while small task-specific modules differ across sides. The latter design appears in semi-Siamese bi-encoders, where the main BERT LLM is kept common for both query and document while separate lightweight fine-tuning modules introduce a limited amount of asymmetry (Jung et al., 2021).

The literature also shows that the term “Siamese” is not always used with identical strictness. StrokeNeXt is described as a Siamese-encoder approach and has two architecturally identical ConvNeXt branches, but the paper is explicit that their parameters are not shared; it therefore fits more precisely under a dual-encoder with Siamese symmetry than a strict tied-weight Siamese network (Ramos et al., 16 Feb 2026). This usage indicates that, in current practice, SDE sometimes names a symmetric two-branch computation pattern even when full weight tying is relaxed.

2. Recurrent architectural patterns

SDE architectures span a broad design space. Some retain the classical late-interaction pattern of independent encoding followed by a single similarity or scoring head. Others introduce repeated cross-branch interaction at multiple scales, or attach decoders, aligners, or diffusion policies to the paired latent streams.

Pattern	Representative paper	Characteristic
Late-interaction retrieval	"Exploring Dual Encoder Architectures for Question Answering" (Dong et al., 2022)	Mean-pooled T5 encodings compared in a shared space
Hierarchical document matching	"Beyond 512 Tokens" (Yang et al., 2020)	Shared long-document towers with sentence-block and document-level Transformers
Cross-scale dense prediction	"dual unet" (Jiang et al., 2022)	Shared-weight dual-stream encoder-decoder with repeated temporal interaction
Coarse-to-fine alignment	"Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision" (Mitchell et al., 2019)	Shared multiscale encoders feeding recursive residual aligners

The simplest SDE pattern remains “encode separately, compare once.” This is the form assumed by standard QA retrieval bi-encoders, by MatchBERT-like document matching systems, and by template-based traffic sign models in which a sample image and a class template are independently mapped into a 256-dimensional feature code and compared with Euclidean distance (Dong et al., 2022, Xi et al., 21 Feb 2025). In these settings, the encoder is the primary object of reuse, and the interaction head is deliberately cheap.

A more interactive pattern appears in dense correspondence and change detection. Dual-UNet is a shared-weight bilateral U-Net network in which encoder features are differenced at every scale, fused back into the branch features, refined by the multiscale differential attention module (MDAM), and then re-compared in the decoder through weighted difference fusion maps (Jiang et al., 2022). SEAMLeSS similarly begins with two shared encoders, but then couples the branches through coarse-to-fine geometric warping and residual displacement estimation at each scale rather than through a single terminal similarity (Mitchell et al., 2019). These examples show that SDE does not imply late interaction.

A third pattern attaches a decoder or generative head to the shared encoder pair. SidAE combines a SimSiam-style tied encoder with a predictor and a decoder that reconstructs the original image from each branch latent (Baier et al., 2023). SiamJEPA replaces the single student encoder of JEPA-style pretraining with two masked student branches that share weights and are regularized by a KL-based cross-view consistency term (Yamada, 4 Jul 2026). VLM-SFD uses a Siamese observation encoder for two target objects and conditions a diffusion model to predict two object-centric motion flows, which are later assigned to robot arms by a VLM-based controller (Chen et al., 16 Jun 2025).

3. Objectives, supervision, and embedding geometry

SDE is an architectural pattern rather than a single optimization recipe. The losses used across the literature range from sampled-softmax ranking objectives to reconstruction, contrastive metric learning, diffusion denoising, and latent prediction.

In retrieval, a standard training objective is in-batch sampled softmax over paired query-document examples, with cosine similarity as the scoring function (Dong et al., 2022). SamToNe modifies this objective by adding same-tower negatives to the denominator, for example terms of the form $e^{\mathtt{sim}(q_i,q_j)/\tau}$, so that a query is pushed away not only from irrelevant documents but also from other queries in the batch (Moiseev et al., 2023). On large SDE QA retrieval models, this raised average $P@1/MRR$ from $44.7/57.5$ to $46.4/58.7$, and on BEIR it improved SDE-XXL average $X^{\text{pre}}$0 from $X^{\text{pre}}$1 to $X^{\text{pre}}$2 (Moiseev et al., 2023).

In self-supervised image representation learning, SidAE uses a convex combination of reconstruction and Siamese alignment,

$X^{\text{pre}}$3

where the Siamese term is SimSiam-style negative cosine similarity and the decoder reconstructs the original image rather than the augmented view (Baier et al., 2023). SiamJEPA uses a JEPA-style latent prediction term over masked regions together with a KL regularizer between branch-conditioned prior and posterior distributions, and reports that stronger Siamese regularization substantially improves early linear-probe accuracy (Yamada, 4 Jul 2026).

In dense prediction, SDE losses are often explicitly imbalance-aware. Dual-UNet uses a batch-balanced contrastive loss with changed and unchanged pixels normalized by their batch frequencies and a margin $X^{\text{pre}}$4, rather than ordinary cross-entropy (Jiang et al., 2022). In time-series augmentation, DTWSSE trains a Siamese encoder with a metric-regression loss

$X^{\text{pre}}$5

so that Euclidean distance in latent space approximates Dynamic Time Warping (DTW), after which interpolation is performed in latent space and decoded back to time series (Yang et al., 2021).

Protein pretraining pushes the idea further away from conventional similarity learning. SiamDiff uses the same sequence-structure encoder on two correlated conformers and optimizes a bidirectional denoising objective that lower-bounds mutual information between diffusion trajectories, rather than using cosine similarity, negatives, or a momentum target encoder (Zhang et al., 2023). A plausible implication is that SDE geometry can be induced by shared denoising semantics, not only by explicit embedding-space contrast.

4. Retrieval and ranking formulations

The retrieval literature provides the clearest explicit definition of SDE. In question answering, a Siamese Dual Encoder shares the token embedder, transformer encoder, and projection layer across the question and answer towers, and the resulting embeddings are compared in a shared space (Dong et al., 2022). That paper reports that SDE performs significantly better than ADE on QA retrieval tasks, and further finds that sharing only the projection layer in an otherwise asymmetric model can make ADE competitive with, or slightly stronger than, SDE on some settings (Dong et al., 2022). The associated t-SNE analysis indicates that shared projection layers help bring question and answer embeddings into a coinciding parameter space.

This retrieval view extends naturally to long-form document matching. SMITH is a Siamese hierarchical Transformer encoder in which each tower independently encodes a long document through sentence-block-level and document-level Transformer stacks, applies L2 normalization, and computes similarity in the resulting document space (Yang et al., 2020). It increases maximum input text length from 512 to 2048 tokens and reports best $X^{\text{pre}}$6 values of $X^{\text{pre}}$7 on Wiki65K and $X^{\text{pre}}$8 on AAN104K (Yang et al., 2020). The key point is architectural: SDE scaling to long documents does not require abandoning independent encoding, but it does require a stronger per-document encoder.

Later retrieval work weakens strict symmetry without fully abandoning it. Semi-Siamese bi-encoders keep the pretrained BERT backbone common while learning tiny query-specific and document-specific lightweight fine-tuning modules, amounting to less than $X^{\text{pre}}$9 different parameters (Jung et al., 2021). On ClueWeb09b, this limited asymmetry is especially helpful, with semi-Siamese ColBERT outperforming both full fine-tuning and Siamese lightweight baselines (Jung et al., 2021). The broad lesson is that SDE and ADE need not be treated as an all-or-nothing dichotomy.

5. Scientific and engineering applications beyond retrieval

Outside retrieval, SDE serves as a general mechanism for comparing structured observations of the same entity across time, condition, or view. In remote sensing change detection, Dual-UNet is a genuine Siamese dual-encoder-decoder architecture with shared weights and recurrent cross-temporal fusion. On the CDD dataset it reports Precision $X^{\text{post}}$0, Recall $X^{\text{post}}$1, $X^{\text{post}}$2 $X^{\text{post}}$3, and IoU $X^{\text{post}}$4, outperforming FC-EF, FC-Siam-diff, FC-Siam-conc, and STANet (Jiang et al., 2022). The Attentive Dual Stream Siamese U-net for flood detection likewise uses two weight-shared encoders on pre- and post-flood Sentinel-1 SAR inputs and reports IoU $X^{\text{post}}$5 and $X^{\text{post}}$6 $X^{\text{post}}$7, a $X^{\text{post}}$8 IoU improvement over a uni-temporal flood detector (Yadav et al., 2022).

Medical imaging supplies both strict and relaxed variants. The lung-nodule growth model based on a Siamese encoder and spatial-temporal mixer processes paired 3D ROIs from consecutive CT scans, extracts $X^{\text{post}}$9, $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$0, and $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$1, and combines them through

$$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$2

with a hierarchical loss that prioritizes growing nodules (Fang et al., 2022). StrokeNeXt, by contrast, uses two ConvNeXt encoders on the same CT image without weight sharing, then fuses the embeddings with a $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$3 Conv1D and a bottleneck $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$4; it reaches accuracies and $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$5-scores of up to $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$6 (Ramos et al., 16 Feb 2026). Together these papers show that current medical literature treats “Siamese” as a continuum from strict tied encoders to symmetric untied dual encoders.

Other domains expose the same architectural motif under different objectives. DTWSSE uses a Siamese encoder plus decoder so that time-series interpolation can occur in a Euclidean latent space aligned with DTW, rather than directly in input space (Yang et al., 2021). IECES uses a sample branch and a template branch with shared Efficient-CNN-based encoders, contrastive loss, and a final SoftMax classifier; after training, the template branch is discarded to improve runtime, yielding a $$F_l^{\text{pre}} = E_l(X^{\text{pre}};\theta), \qquad F_l^{\text{post}} = E_l(X^{\text{post}};\theta),$$7M-parameter real-time traffic sign recognizer (Xi et al., 21 Feb 2025). In robotics, VLM-SFD employs a Siamese Flow Diffusion Network that embeds two target objects into a shared latent space and predicts two motion flows for later arm assignment (Chen et al., 16 Jun 2025). In protein modeling, SiamDiff uses two diffusion-corrupted conformers of the same protein and a shared encoder to maximize information useful for denoising both sequence and structure (Zhang et al., 2023).

6. Variants, misconceptions, and limitations

A common misconception is that SDE always means “encode each input independently and compare once at the end.” Dual-UNet and SEAMLeSS are clear counterexamples: both begin with shared encoders, but both introduce repeated cross-branch interaction during refinement, whether by encoder-scale difference maps, attention, and decoder-side fusion, or by multiscale warping and residual displacement prediction (Jiang et al., 2022, Mitchell et al., 2019). In these systems, the Siamese property guarantees feature comparability, but the downstream computation is deeply interactive.

A second misconception is that “Siamese” always implies complete weight sharing. That is true for the strict SDE definition used in QA retrieval, in long-document matching, and in classical change detection (Dong et al., 2022, Yang et al., 2020). It is not true for all contemporary usage. StrokeNeXt explicitly uses untied but symmetric branches, while semi-Siamese bi-encoders share the backbone and introduce only a small amount of query- or document-specific adaptation (Ramos et al., 16 Feb 2026, Jung et al., 2021). The literature therefore supports a narrower definition, in which SDE means full parameter tying, and a broader usage, in which Siamese names branch symmetry plus strong representational coupling.

The empirical record also shows practical limitations. Some application papers leave crucial implementation details underspecified, which can complicate faithful reproduction; Dual-UNet is explicit about several modules but leaves parts of the attention and fusion formulas unspecified (Jiang et al., 2022). Some loss modifications are data-regime dependent: SamToNe improves SDE retrieval on average, but document-side same-tower negatives can hurt when many training examples share duplicate documents (Moiseev et al., 2023). A plausible implication is that SDE behavior is governed not only by tower sharing, but by the interaction between sharing pattern, loss geometry, and the structure of paired supervision.

Taken together, the recent literature presents SDE not as a single model class with a fixed scoring rule, but as a reusable architectural prior. Its defining commitment is that paired inputs should first be expressed through a common encoder geometry; what happens after that may be cosine retrieval, multiscale differencing, cross-view denoising, hierarchical decoding, or diffusion-based policy generation.