---
title: Encoder-Decoder Siamese Architecture (E2E-R)
url: https://www.emergentmind.com/topics/encoder-decoder-siamese-architecture-e2e-r
type: topic
---

# Encoder-Decoder Siamese Architecture (E2E-R)

Searching arXiv for the cited works and adjacent uses of encoder–decoder Siamese architectures.
arXiv search: "A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation"
Encoder–Decoder Siamese Architecture (E2E-R) denotes a family of end-to-end models in which two tied or explicitly aligned input branches produce comparable latent representations that are consumed by a decoder, aligner, or task head. In the cited literature, the acronym itself is usually absent; instead, individual papers describe end-to-end trainable Siamese encoder–decoder systems for pivot-based sequence generation, change detection, image registration, detection pretraining, video object segmentation, 2D-to-3D lifting, self-supervised representation learning, and time-series augmentation. The recurring design problem is how the twin branches should be coupled—by weight sharing, correlation, cross-attention, warping, or predictor-based consistency—and how the decoder should transform the resulting paired representation into a sequence, mask, deformation field, pose, reconstruction, or synthetic sample [1606.04754][2201.01293][1904.02643][2303.18144][2112.13983][1809.07217][2304.02549][2108.09885].

## 1. Conceptual scope and defining properties

The central structural trait of E2E-R is the joint treatment of paired inputs together with a downstream decoder. In classical Siamese usage, the twin branches share parameters. Several E2E-R realizations follow that pattern directly: ChangeFormer uses a hierarchical transformer encoder with shared weights for pre-change and post-change images; SEAMLeSS uses identical-weight convolutional encoders for source and target images; SITVOS uses a weight-shared ResNet50 backbone for past and current frames; Siamese DETR shares the backbone, Transformer encoder–decoder, and object queries across views; the pose-estimation model shares the encoder–decoder across camera views; SidAE shares encoder, predictor, and decoder across augmented views; and DTWSSE uses a shared-weight encoder for time-series pairs [2201.01293][1904.02643][2112.13983][2303.18144][1809.07217][2304.02549][2108.09885].

A significant variant departs from strict parameter sharing. The correlational encoder–decoder for pivot-based generation uses two separate encoders, $f_X(\cdot;\theta_X)$ and $f_Z(\cdot;\theta_Z)$, and aligns them by a correlation-based regularizer rather than by tying weights. Its shared-space variables are
$$
h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,
$$
with standardization
$$
s(h)=\frac{h-\mu}{\sigma}.
$$
This yields what the paper characterizes as a “jointly aligned” rather than “weight-shared” Siamese design, while still fitting an encoder–decoder representation-alignment interpretation of E2E-R [1606.04754].

Several works explicitly note that they do not define the acronym “E2E-R” themselves. In those cases, the label functions as a retrospective umbrella over architectures that are simultaneously encoder–decoder, Siamese in branch structure or representation, and trained end-to-end on the task objective rather than as disjoint feature extractors followed by separate downstream modules. Taken together, these papers indicate that E2E-R is best understood as an architectural pattern rather than a single canonical topology [2201.01293][2303.18144][1904.02643][2112.13983][2304.02549][2108.09885].

## 2. Structural variants across the literature

The architectural family is heterogeneous in the form of the encoder, in the mechanism used to compare or align the two branches, and in the meaning of the “decoder.”

| Realization | Siamese coupling | Decoder or output role |
|---|---|---|
| "A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation" [1606.04754] | Separate encoders aligned by correlation in a shared interlingua-like space | RNN decoder generates $Y$ from the common representation |
| "A Transformer-Based Siamese Network for Change Detection" [2201.01293] | Shared hierarchical transformer encoder on pre/post images | MLP decoder fuses multi-scale difference maps into a binary change mask |
| "Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision" [1904.02643] | Shared multiscale convolutional encoders | Hierarchical aligners refine a dense displacement field |
| "Siamese DETR" [2303.18144] | Shared backbone, Transformer, and object queries across views | Decoder with Multi-View Cross-Attention performs localization and region-level semantics |
| "Siamese Network with Interactive Transformer for Video Object Segmentation" [2112.13983] | Shared ResNet50 backbone for past/current frames | Transformer decoder and segmentation decoder retrieve the current-frame target |
| "3D Human Pose Estimation with Siamese Equivariant Embedding" [1809.07217] | Shared encoder–decoder across camera-view pairs | Decoder lifts a rotation-equivariant embedding to 3D joints |
| "Self-Supervised Siamese Autoencoders" [2304.02549] | Shared encoder, predictor, and decoder across two augmentations | Decoder reconstructs the clean image from each latent code |
| "DTWSSE: Data Augmentation with a Siamese Encoder for Time Series" [2108.09885] | Shared encoder maps time-series pairs into Euclidean deep feature space | Decoder reconstructs sequences and enables latent interpolation |

Within this broad template, the interaction mechanism is the most task-specific component. ChangeFormer forms a learned difference at each scale by concatenating $F^i_{\mathrm{pre}}$ and $F^i_{\mathrm{post}}$ and applying $\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$ instead of using a fixed absolute difference. SEAMLeSS warps one branch’s multiscale features with the current displacement estimate before residual prediction. Siamese DETR modifies decoder cross-attention so that shared queries are conditioned on region features from one view and attend to encoder outputs of the other view. SITVOS inserts a Feature Interaction Module between memory encoding and current-frame decoding, with bidirectional cross-attention and mask gating. SidAE inserts a predictor head between encoder and Siamese loss, while still decoding both latents back to the clean image. DTWSSE learns a Euclidean latent geometry that approximates Dynamic Time Warping, then decodes interpolated latent points back into time series. This suggests that the “Siamese” component in E2E-R is less a single operator than a design principle for enforcing comparability between paired streams.

## 3. Training objectives and optimization regimes

E2E-R systems are unified less by a single loss than by end-to-end optimization over coupled encoder and decoder objectives. In the correlational encoder–decoder, the training data are split into $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$ for $X$–$Z$ pairs and $D_2=\{(z_k,y_k)\}_{k=1}^{N_2}$ for $Z$–$Y$ pairs. The model alternates mini-batches from the two datasets and minimizes
$$
L_{\text{total}}(\theta)=J_{\text{corr}}(\theta)+L_{\text{dec}}(\theta),
$$
where
$$
J_{\text{corr}}(\theta)=-\lambda\sum_{i=1}^N s(h_X(x_i))\,s(h_Z(z_i))^T
$$
and
$$
L_{\text{dec}}(\theta)=-\frac{1}{N_2}\sum_{k=1}^{N_2}\sum_{t=1}^{L_k}\log p(y_{k,t}\mid y_{k,<t},h_Z(z_k)).
$$
Teacher forcing is used, $\lambda\in[0.1,1.0]$ balances alignment and decodability, and Adam is the optimizer [1606.04754].

In change detection, the end-to-end loss is pixel-wise binary Cross-Entropy,
$$
L_{\mathrm{CE}}=-\frac{1}{N}\sum_i \left[y_i\log(\hat y_i)+(1-y_i)\log(1-\hat y_i)\right],
$$
applied to a two-class mask after the MLP decoder. The network is trained in PyTorch with AdamW, weight decay $0.01$, $\beta=(0.9,0.999)$, learning rate $1\times10^{-4}$ linearly decayed to $0$ over $200$ epochs, and batch size $16$ [2201.01293].

SEAMLeSS uses self-supervision rather than explicit transform labels. At MIP level $n$, the stage loss is
$$
L^{(n)}=L_{\text{data}}^{(n)}+L_{\text{smooth,masked}}^{(n)},
$$
with
$$
L_{\text{data}}^{(n)}=\sum_x \|T_n(x)-W(S_n,u^n)(x)\|_2^2
$$
and
$$
L_{\text{smooth,masked}}^{(n)}=\lambda\sum_{\langle x,x'\rangle} M(x)M(x')\|u^n(x)-u^n(x')\|_2^2,
$$
where $\lambda=0.1$. Training is hierarchical from coarse to fine, two epochs per stage, with the encoders always trained and the last epoch in each stage jointly training the active and coarser aligners [1904.02643].

Siamese DETR combines localization and representation learning. Its total pretraining objective is
$$
\mathcal{L}=\lambda_0\mathcal{L}_r+\lambda_1\mathcal{L}_g+\lambda_2\mathcal{L}_{\text{loc}},
$$
where $\mathcal{L}_{\text{loc}}$ is symmetric box regression with Hungarian matching and DETR-style $L_1+\mathrm{GIoU}$ terms, $\mathcal{L}_g$ is BYOL-style global negative cosine similarity with stop-gradient, and $\mathcal{L}_r$ is normalized $\ell_2$ distance between decoded region features and target crop-level features. The reported hyperparameters are $\lambda_2=1.0$ in all setups, $\lambda_0=3,\lambda_1=10$ on ImageNet, and $\lambda_0=0.3,\lambda_1=3$ on COCO [2303.18144].

SITVOS minimizes a two-class pixel-wise cross-entropy loss,
$$
L=\lambda_{\mathrm{ce}}L_{\mathrm{ce}},\qquad \lambda_{\mathrm{ce}}=1,
$$
over the segmentation decoder’s logits, while the recurrent element is handled operationally through memory updates rather than through an explicit recurrent cell [2112.13983]. The 3D pose model minimizes
$$
\ell=\ell_{3D}^{(1)}+\ell_{3D}^{(2)}+\lambda_2\ell_S,
$$
where $\ell_S$ enforces rotation-equivariant consistency between the paired latent embeddings using calibrated relative camera rotations $R_2R_1^{-1}$, and each branch also carries a 3D regression loss [1809.07217].

Self-supervised variants replace task supervision with consistency and reconstruction. SidAE uses
$$
L_{\mathrm{SidAE}}=wL_{\mathrm{dae}}+(1-w)L_{\mathrm{si}},
$$
where $L_{\mathrm{dae}}$ is MSE reconstruction of the clean image from each augmented view’s latent code and $L_{\mathrm{si}}$ is a SimSiam-style negative cosine similarity with stop-gradient [2304.02549]. DTWSSE trains sequentially rather than jointly: first the Siamese encoder is optimized to regress latent Euclidean distance to DTW,
$$
\mathcal{L}_E=\frac{1}{|\mathcal D|}\sum_i \left(\|h_i^1-h_i^2\|_2-y_i\right)^2,
$$
then the decoder is trained with reconstruction loss
$$
\mathcal{L}_D=\frac{1}{2|\mathcal D|}\sum_i\|S_i^1-\hat S_i^1\|_2^2+\frac{1}{2|\mathcal D|}\sum_i\|S_i^2-\hat S_i^2\|_2^2.
$$
A plausible implication is that “end-to-end” in the E2E-R label should be read structurally rather than as a requirement of a single monolithic objective, because the literature includes both simultaneous and staged optimization [2108.09885].

## 4. Functional realizations in different domains

In pivot-based sequence generation, E2E-R addresses the case in which no $X$–$Y$ pairs exist, but $X$–$Z$ and $Z$–$Y$ pairs do. The correlational encoder–decoder learns a shared interlingua-like space for $X$ and $Z$ and decodes $Y$ from that space. At test time it encodes $x\in X$ with $f_X$ and decodes $y\in Y$ directly, without explicitly generating the pivot $z$. For transliteration, the encoders are character-level RNNs without attention; for captioning, the image encoder projects a precomputed CNN feature from a fully connected layer into the shared space, and the decoder is a word-level RNN [1606.04754].

In remote sensing change detection, the Siamese encoder–decoder serves pixel-wise segmentation. ChangeFormer takes a pair of co-registered images and feeds them through identical hierarchical transformer encoders. Each stage yields a multi-scale feature map, and a Difference Module learns the interaction between timepoints through concatenation followed by $\mathrm{Conv2D}\rightarrow\mathrm{ReLU}\rightarrow\mathrm{BatchNorm}$. An MLP decoder unifies channels, upsamples all scales to a common resolution, concatenates them, and produces a two-class change mask. The encoder uses sequence reduction to lower attention complexity from $O(N^2)$ to $O(N^2/R)$ and replaces fixed positional embeddings with a learned, resolution-agnostic MLP plus depth-wise convolution module [2201.01293].

In registration, the decoder role is played by a hierarchy of aligners that progressively refine a dense vector field. SEAMLeSS forms multiscale feature hierarchies with Siamese convolutional encoders, then performs coarse-to-fine residual alignment. At each level, the current displacement estimate is upsampled, the source features are warped by bilinear interpolation, and a 5-layer $7\times7$ convolutional aligner predicts a residual displacement. The refined field is passed to the next finer level, and the final output is a full-resolution warp $F(r)=r+D(r)$ [1904.02643].

In detection pretraining, the encoder–decoder structure is internal to DETR. Siamese DETR constructs two augmented views with IoU-constrained overlap, extracts image-level features with a frozen SwAV-pretrained ResNet-50, encodes them with a shared 6-layer Transformer encoder, and decodes them with a shared 6-layer Transformer decoder. The distinctive operator is Multi-View Cross-Attention:
$$
\hat h_2=\mathrm{MHA}(f_q(z_1+q),f_k(E_2),f_v(E_2)),
$$
and symmetrically for the other direction. Localization is therefore learned as cross-view region retrieval rather than conventional image-level invariance alone [2303.18144].

In semi-supervised video object segmentation, SITVOS uses a Siamese ResNet50 backbone for memory frames and the current frame, a transformer encoder over flattened past-frame tokens, a transformer decoder over current-frame tokens, and a Feature Interaction Module that performs mutual cross-attention between the two streams. Memory masks are embedded by a ResNet18 whose first convolution accepts a single channel, and the interaction module uses the mask-filtered memory $M_x=M_E\odot M_{\mathrm{ori}}$ to suppress background in the decoder-side cross-attention. The output is a current-frame mask obtained by a segmentation decoder with refinement modules and skip connections [2112.13983].

The same family extends beyond dense prediction. In monocular 3D human pose estimation, the Siamese branches process paired 2D keypoint detections from calibrated cameras, the encoder maps them into a latent $h\in\mathbb{R}^{3\times M}$ of $M$ normalized 3D vectors, and the decoder lifts the latent code to 3D joints while a Siamese equivariance loss enforces $Rh^{(1)}\approx h^{(2)}$ for the known relative rotation $R$ [1809.07217]. In self-supervised learning, SidAE couples a shared ResNet-18 encoder plus projector with a predictor head and a shared decoder built from a fully connected layer and five transposed convolutions to simultaneously impose view consistency and denoising reconstruction [2304.02549]. In time-series augmentation, DTWSSE trains a Siamese encoder so that latent Euclidean distance approximates DTW and then uses the decoder to map interpolated latent points back to the time-series space, effectively replacing raw-space SMOTE interpolation with DTW-aligned latent interpolation [2108.09885].

## 5. Reported empirical behavior

The empirical record is mixed and task-dependent rather than uniformly favorable. In bridge transliteration, the correlational encoder–decoder surpasses the two-stage encoder–decoder in $6/12$ language pairs and exceeds the PBSMT baseline in $11/12$ pairs, with a particularly large gain for Ma$\rightarrow$Hi: $59.0$ versus $45.8$ for the two-stage encoder–decoder. By contrast, in bridge captioning the correlational model trails both strong baselines, scoring BLEU-4 $=12.6$, ROUGE-L $=34.3$, and CIDEr $=29.8$, versus BLEU-4 $=16.6$, ROUGE-L $=39.5$, and CIDEr $=49.1$ for the two-stage pipeline. The model nevertheless produces plausible French captions from images without any Image–French training pairs [1606.04754].

For remote sensing change detection, ChangeFormer reports on LEVIR-CD test data $F1=90.40\%$, $\mathrm{IoU}=82.48\%$, and $\mathrm{OA}=99.04\%$, compared with BIT at $F1=89.31\%$, $\mathrm{IoU}=80.68\%$, and $\mathrm{OA}=98.92\%$, corresponding to $+1.2\%$ F1, $+2.2\%$ IoU, and $+0.12\%$ OA over the previous state of the art. On DSIFN-CD, the gains are larger: $F1=86.67\%$, $\mathrm{IoU}=76.48\%$, and $\mathrm{OA}=95.56\%$, versus BIT at $69.26\%$, $52.97\%$, and $89.41\%$, for reported improvements of $+20.0\%$ F1, $+44.3\%$ IoU, and $+6.4\%$ OA [2201.01293].

For self-supervised alignment in serial-section electron microscopy, SEAMLeSS improves chunked Pearson correlation relative to several baselines. On Pinky40-prealigned images, reported CPC means are None $0.224$, FlowNet $0.417$, SPyNet $0.497$, Alembic $0.528$, and SEAMLeSS $0.545$. The paper further reports better behavior in the $95$th and $99$th percentile tails and emphasizes that Siamese encoding avoids the worst coarse-level failures seen with SPyNet’s raw image pyramids [1904.02643].

In detection pretraining, Siamese DETR consistently improves transfer across DETR variants. On COCO, Vanilla DETR improves from AP $39.5$ from scratch to AP $42.0$, Conditional DETR from $41.1$ to $43.0$, Deformable DETR single-scale from $40.3$ to $42.1$, and Deformable DETR multi-scale from $45.5$ to $46.3$. On PASCAL VOC, Deformable DETR multi-scale improves from AP $56.1$ from scratch to $61.2$, and Conditional DETR improves from $49.9$ to $58.1$. The gains are larger at AP75 than AP50, which the paper interprets as stronger localization priors. The method also reports AR@100 $=30.7$, compared with DETReg at $26.5$, UP-DETR at $24.0$, and random initialization at $0.5$ [2303.18144].

SITVOS reports DAVIS 2017-Val $J\&F=83.5\%$, $J_M=80.4\%$, $F_M=86.5\%$, and $11.8$ FPS; DAVIS 2016-Val $J\&F=90.5\%$, $J_M=89.5\%$, and $F_M=91.4\%$; and YouTube-VOS Val overall $G=81.3$, with $J_s=79.9$, $J_u=76.4$, $F_s=84.3$, and $F_u=84.4$. The Feature Interaction Module contributes $+1.3\%$ on DAVIS16 and $+1.5\%$ on DAVIS17 relative to the ablated model without FIM [2112.13983].

For camera-robust 3D lifting, the equivariant Siamese model reports a Protocol #3 cross-camera average MPJPE of $65.8$ mm on Human3.6M when synthetic cameras are used every $15^\circ$, outperforming PoseGrammar with the same augmentation at $67.8$ mm. On Protocol #1, the gain is smaller, $61.1$ mm versus $62.9$ mm, consistent with the observation that all training viewpoints are already present. Ablations show that removing the Siamese loss degrades Protocol #3 performance to $71.1$ mm, and removing rotation augmentation degrades it to $81.0$ mm [1809.07217].

In self-supervised image representation learning, SidAE consistently exceeds both SimSiam and the denoising autoencoder baseline. On CIFAR-10 with $1\%$ labels after $200$ pretraining epochs and a frozen encoder, SidAE reaches $51.67\,(0.28)$ accuracy, compared with SimSiam at $44.06\,(3.74)$ and AE at $40.67\,(0.25)$. On Fashion-MNIST with $1\%$ labels, the corresponding figures are $81.62\,(0.22)$, $77.84\,(0.19)$, and $80.20\,(0.31)$; on MNIST they are $94.65\,(0.49)$, $79.42\,(5.15)$, and $88.41\,(0.46)$. On STL-10 pretraining transferred to CIFAR-10, SidAE reports $66.20\,(0.31)$ versus $56.64\,(0.25)$ for SimSiam and $57.08\,(3.16)$ for AE [2304.02549].

In time-series augmentation, DTWSSE generally outperforms SMOTE and related ablations. On DistalPhalanxTW with a fully connected autoencoder, Top-1 accuracy improves from $64.75\%$ without augmentation and $69.06\%$ with SMOTE to $74.10\%$ with DTWSSE; Worst accuracy improves from $51.80\%$ and $62.47\%$ to $63.31\%$; and Average accuracy improves from $59.44\%$ and $65.20\%$ to $67.72\%$. On balanced datasets, DTWSSE(CNN) reaches Top-1 $=99.13\%$ and Average $=98.25\%$ on Chinatown, and Top-1 $=97.13\%$ and Average $=96.26\%$ on MelbournePedestrian. On Libras, DTWSSE(DNN) reports Average $=93.40\%$, compared with $91.67\%$ for SMOTE and $92.36\%$ without augmentation [2108.09885].

## 6. Limitations, misconceptions, and extension points

A frequent misconception is that any Siamese encoder–decoder must share encoder weights. The correlational encoder–decoder contradicts that assumption directly: its encoders are separate networks with $\theta_X\neq\theta_Z$, and the shared structure is enforced in representation space by correlation rather than in parameter space. A second misconception is that the decoder must be generative in the narrow sense of text or image reconstruction. In this literature, the decoder may instead be an MLP segmentation head, a hierarchical aligner, a DETR decoder with cross-view attention, or a 3D pose regressor [1606.04754][2201.01293][1904.02643][2303.18144][1809.07217].

The limitations are equally domain-specific. The correlational encoder–decoder is sensitive to pivot quality and domain, requires tuning of $\lambda$, and lags strong baselines on captioning; the paper notes that attention and richer conditioning may be needed for complex multimodal semantics. ChangeFormer assumes co-registered input pairs and can degrade under misregistration, while still facing transformer memory–computation trade-offs despite sequence reduction. SEAMLeSS requires smoothness regularization for self-supervised training and needs masking near discontinuities such as cracks and folds to avoid oversmoothing true non-smooth transforms [1606.04754][2201.01293][1904.02643].

Further limitations arise from the way paired correspondence is constructed. Siamese DETR depends on view overlap and the quality of object proposals, although it is reported to remain robust across proposal sources and to benefit from EdgeBoxes. SITVOS remains challenged by severe background distractors that closely resemble the target and by extremely fast motion, even though spatio-temporal aggregation and mask gating improve robustness to occlusion, deformation, and disappearance. The pose-estimation model depends on calibrated relative camera rotations and still exhibits failure modes in sitting poses, which the paper attributes to dataset bias toward standing [2303.18144][2112.13983][1809.07217].

Self-supervised and augmentation-oriented E2E-R models introduce another set of trade-offs. SidAE requires tuning the interpolation weight $w$: large $w$ moves the model toward pixel-level reconstruction and can weaken invariance, while the empirical results indicate that lower $w$ values are often preferable on harder datasets such as CIFAR-10 and STL-10. DTWSSE inherits the $O(L^2)$ cost of DTW for neighbor selection, trains encoder and decoder sequentially rather than jointly, and shows dataset dependence in the choice between CNN and fully connected encoder–decoder variants. The paper also reports that a naive autoencoder without DTW-aligned latent geometry can sharply degrade worst-case accuracy on some datasets [2304.02549][2108.09885].

The literature also suggests several extension points. The pivot-generation work explicitly proposes attention, contrastive correlations, or Deep CCA as ways to strengthen shared-space learning while preserving end-to-end decodability. ChangeFormer’s analysis points to domain adaptation under sensor and seasonal shift. Siamese DETR discusses extensions to segmentation, panoptic detection, video detection, and unified end-to-end pretraining of both backbone and Transformer. DTWSSE notes that warping windows, lower bounds, joint objectives, or additional latent regularization could reduce cost or improve stability. These proposals reinforce a general conclusion: E2E-R is not a fixed recipe but a reusable architectural schema in which paired encoders and a decoder are optimized together so that inter-branch comparability directly supports the downstream prediction task [1606.04754][2201.01293][2303.18144][2108.09885].

Source: https://www.emergentmind.com/topics/encoder-decoder-siamese-architecture-e2e-r