Encoder-Decoder Siamese Architecture (E2E-R)

• Encoder–Decoder Siamese Architecture (E2E-R) is a design pattern that uses paired encoders to produce comparable latent representations fed into a unified decoder.
• It employs various coupling methods like weight sharing, correlation, and cross-attention to align dual input streams for tasks such as change detection, image registration, and pose estimation.
• Empirical studies highlight performance gains and trade-offs across applications, emphasizing the adaptability and limitations of different alignment strategies.

Searching arXiv for the cited works and adjacent uses of encoder–decoder Siamese architectures. arXiv search: "A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation" Encoder–Decoder Siamese Architecture (E2E-R) denotes a family of end-to-end models in which two tied or explicitly aligned input branches produce comparable latent representations that are consumed by a decoder, aligner, or task head. In the cited literature, the acronym itself is usually absent; instead, individual papers describe end-to-end trainable Siamese encoder–decoder systems for pivot-based sequence generation, change detection, image registration, detection pretraining, video object segmentation, 2D-to-3D lifting, self-supervised representation learning, and time-series augmentation. The recurring design problem is how the twin branches should be coupled—by weight sharing, correlation, cross-attention, warping, or predictor-based consistency—and how the decoder should transform the resulting paired representation into a sequence, mask, deformation field, pose, reconstruction, or synthetic sample (Saha et al., 2016, Bandara et al., 2022, Mitchell et al., 2019, Chen et al., 2023, Lan et al., 2021, Véges et al., 2018, Baier et al., 2023, Yang et al., 2021).

1. Conceptual scope and defining properties

The central structural trait of E2E-R is the joint treatment of paired inputs together with a downstream decoder. In classical Siamese usage, the twin branches share parameters. Several E2E-R realizations follow that pattern directly: ChangeFormer uses a hierarchical transformer encoder with shared weights for pre-change and post-change images; SEAMLeSS uses identical-weight convolutional encoders for source and target images; SITVOS uses a weight-shared ResNet50 backbone for past and current frames; Siamese DETR shares the backbone, Transformer encoder–decoder, and object queries across views; the pose-estimation model shares the encoder–decoder across camera views; SidAE shares encoder, predictor, and decoder across augmented views; and DTWSSE uses a shared-weight encoder for time-series pairs (Bandara et al., 2022, Mitchell et al., 2019, Lan et al., 2021, Chen et al., 2023, Véges et al., 2018, Baier et al., 2023, Yang et al., 2021).

A significant variant departs from strict parameter sharing. The correlational encoder–decoder for pivot-based generation uses two separate encoders, $f_X(\cdot;\theta_X)$ and $f_Z(\cdot;\theta_Z)$, and aligns them by a correlation-based regularizer rather than by tying weights. Its shared-space variables are

$$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$

with standardization

$s(h)=\frac{h-\mu}{\sigma}.$

This yields what the paper characterizes as a “jointly aligned” rather than “weight-shared” Siamese design, while still fitting an encoder–decoder representation-alignment interpretation of E2E-R (Saha et al., 2016).

Several works explicitly note that they do not define the acronym “E2E-R” themselves. In those cases, the label functions as a retrospective umbrella over architectures that are simultaneously encoder–decoder, Siamese in branch structure or representation, and trained end-to-end on the task objective rather than as disjoint feature extractors followed by separate downstream modules. Taken together, these papers indicate that E2E-R is best understood as an architectural pattern rather than a single canonical topology (Bandara et al., 2022, Chen et al., 2023, Mitchell et al., 2019, Lan et al., 2021, Baier et al., 2023, Yang et al., 2021).

2. Structural variants across the literature

The architectural family is heterogeneous in the form of the encoder, in the mechanism used to compare or align the two branches, and in the meaning of the “decoder.”

Realization	Siamese coupling	Decoder or output role
"A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation" (Saha et al., 2016)	Separate encoders aligned by correlation in a shared interlingua-like space	RNN decoder generates $Y$ from the common representation
"A Transformer-Based Siamese Network for Change Detection" (Bandara et al., 2022)	Shared hierarchical transformer encoder on pre/post images	MLP decoder fuses multi-scale difference maps into a binary change mask
"Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision" (Mitchell et al., 2019)	Shared multiscale convolutional encoders	Hierarchical aligners refine a dense displacement field
"Siamese DETR" (Chen et al., 2023)	Shared backbone, Transformer, and object queries across views	Decoder with Multi-View Cross-Attention performs localization and region-level semantics
"Siamese Network with Interactive Transformer for Video Object Segmentation" (Lan et al., 2021)	Shared ResNet50 backbone for past/current frames	Transformer decoder and segmentation decoder retrieve the current-frame target
"3D Human Pose Estimation with Siamese Equivariant Embedding" (Véges et al., 2018)	Shared encoder–decoder across camera-view pairs	Decoder lifts a rotation-equivariant embedding to 3D joints
"Self-Supervised Siamese Autoencoders" (Baier et al., 2023)	Shared encoder, predictor, and decoder across two augmentations	Decoder reconstructs the clean image from each latent code
"DTWSSE: Data Augmentation with a Siamese Encoder for Time Series" (Yang et al., 2021)	Shared encoder maps time-series pairs into Euclidean deep feature space	Decoder reconstructs sequences and enables latent interpolation

Within this broad template, the interaction mechanism is the most task-specific component. ChangeFormer forms a learned difference at each scale by concatenating $F^i_{\mathrm{pre}}$ and $F^i_{\mathrm{post}}$ and applying $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$ instead of using a fixed absolute difference. SEAMLeSS warps one branch’s multiscale features with the current displacement estimate before residual prediction. Siamese DETR modifies decoder cross-attention so that shared queries are conditioned on region features from one view and attend to encoder outputs of the other view. SITVOS inserts a Feature Interaction Module between memory encoding and current-frame decoding, with bidirectional cross-attention and mask gating. SidAE inserts a predictor head between encoder and Siamese loss, while still decoding both latents back to the clean image. DTWSSE learns a Euclidean latent geometry that approximates Dynamic Time Warping, then decodes interpolated latent points back into time series. This suggests that the “Siamese” component in E2E-R is less a single operator than a design principle for enforcing comparability between paired streams.

3. Training objectives and optimization regimes

E2E-R systems are unified less by a single loss than by end-to-end optimization over coupled encoder and decoder objectives. In the correlational encoder–decoder, the training data are split into $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$ for $X$–$f_Z(\cdot;\theta_Z)$0 pairs and $f_Z(\cdot;\theta_Z)$1 for $f_Z(\cdot;\theta_Z)$2–$f_Z(\cdot;\theta_Z)$3 pairs. The model alternates mini-batches from the two datasets and minimizes

$f_Z(\cdot;\theta_Z)$4

where

$f_Z(\cdot;\theta_Z)$5

and

$f_Z(\cdot;\theta_Z)$6

Teacher forcing is used, $f_Z(\cdot;\theta_Z)$7 balances alignment and decodability, and Adam is the optimizer (Saha et al., 2016).

In change detection, the end-to-end loss is pixel-wise binary Cross-Entropy,

$f_Z(\cdot;\theta_Z)$8

applied to a two-class mask after the MLP decoder. The network is trained in PyTorch with AdamW, weight decay $f_Z(\cdot;\theta_Z)$9, $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$0, learning rate $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$1 linearly decayed to $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$2 over $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$3 epochs, and batch size $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$4 (Bandara et al., 2022).

SEAMLeSS uses self-supervision rather than explicit transform labels. At MIP level $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$5, the stage loss is

$$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$6

with

$$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$7

and

$$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$8

where $$h_X=f_X(x;\theta_X)\in\mathbb{R}^d,\qquad h_Z=f_Z(z;\theta_Z)\in\mathbb{R}^d,$$9. Training is hierarchical from coarse to fine, two epochs per stage, with the encoders always trained and the last epoch in each stage jointly training the active and coarser aligners (Mitchell et al., 2019).

Siamese DETR combines localization and representation learning. Its total pretraining objective is

$s(h)=\frac{h-\mu}{\sigma}.$0

where $s(h)=\frac{h-\mu}{\sigma}.$1 is symmetric box regression with Hungarian matching and DETR-style $s(h)=\frac{h-\mu}{\sigma}.$2 terms, $s(h)=\frac{h-\mu}{\sigma}.$3 is BYOL-style global negative cosine similarity with stop-gradient, and $s(h)=\frac{h-\mu}{\sigma}.$4 is normalized $s(h)=\frac{h-\mu}{\sigma}.$5 distance between decoded region features and target crop-level features. The reported hyperparameters are $s(h)=\frac{h-\mu}{\sigma}.$6 in all setups, $s(h)=\frac{h-\mu}{\sigma}.$7 on ImageNet, and $s(h)=\frac{h-\mu}{\sigma}.$8 on COCO (Chen et al., 2023).

SITVOS minimizes a two-class pixel-wise cross-entropy loss,

$s(h)=\frac{h-\mu}{\sigma}.$9

over the segmentation decoder’s logits, while the recurrent element is handled operationally through memory updates rather than through an explicit recurrent cell (Lan et al., 2021). The 3D pose model minimizes

$Y$0

where $Y$1 enforces rotation-equivariant consistency between the paired latent embeddings using calibrated relative camera rotations $Y$2, and each branch also carries a 3D regression loss (Véges et al., 2018).

Self-supervised variants replace task supervision with consistency and reconstruction. SidAE uses

$Y$3

where $Y$4 is MSE reconstruction of the clean image from each augmented view’s latent code and $Y$5 is a SimSiam-style negative cosine similarity with stop-gradient (Baier et al., 2023). DTWSSE trains sequentially rather than jointly: first the Siamese encoder is optimized to regress latent Euclidean distance to DTW,

$Y$6

then the decoder is trained with reconstruction loss

$Y$7

A plausible implication is that “end-to-end” in the E2E-R label should be read structurally rather than as a requirement of a single monolithic objective, because the literature includes both simultaneous and staged optimization (Yang et al., 2021).

4. Functional realizations in different domains

In pivot-based sequence generation, E2E-R addresses the case in which no $Y$8–$Y$9 pairs exist, but $F^i_{\mathrm{pre}}$0–$F^i_{\mathrm{pre}}$1 and $F^i_{\mathrm{pre}}$2–$F^i_{\mathrm{pre}}$3 pairs do. The correlational encoder–decoder learns a shared interlingua-like space for $F^i_{\mathrm{pre}}$4 and $F^i_{\mathrm{pre}}$5 and decodes $F^i_{\mathrm{pre}}$6 from that space. At test time it encodes $F^i_{\mathrm{pre}}$7 with $F^i_{\mathrm{pre}}$8 and decodes $F^i_{\mathrm{pre}}$9 directly, without explicitly generating the pivot $F^i_{\mathrm{post}}$0. For transliteration, the encoders are character-level RNNs without attention; for captioning, the image encoder projects a precomputed CNN feature from a fully connected layer into the shared space, and the decoder is a word-level RNN (Saha et al., 2016).

In remote sensing change detection, the Siamese encoder–decoder serves pixel-wise segmentation. ChangeFormer takes a pair of co-registered images and feeds them through identical hierarchical transformer encoders. Each stage yields a multi-scale feature map, and a Difference Module learns the interaction between timepoints through concatenation followed by $F^i_{\mathrm{post}}$1. An MLP decoder unifies channels, upsamples all scales to a common resolution, concatenates them, and produces a two-class change mask. The encoder uses sequence reduction to lower attention complexity from $F^i_{\mathrm{post}}$2 to $F^i_{\mathrm{post}}$3 and replaces fixed positional embeddings with a learned, resolution-agnostic MLP plus depth-wise convolution module (Bandara et al., 2022).

In registration, the decoder role is played by a hierarchy of aligners that progressively refine a dense vector field. SEAMLeSS forms multiscale feature hierarchies with Siamese convolutional encoders, then performs coarse-to-fine residual alignment. At each level, the current displacement estimate is upsampled, the source features are warped by bilinear interpolation, and a 5-layer $F^i_{\mathrm{post}}$4 convolutional aligner predicts a residual displacement. The refined field is passed to the next finer level, and the final output is a full-resolution warp $F^i_{\mathrm{post}}$5 (Mitchell et al., 2019).

In detection pretraining, the encoder–decoder structure is internal to DETR. Siamese DETR constructs two augmented views with IoU-constrained overlap, extracts image-level features with a frozen SwAV-pretrained ResNet-50, encodes them with a shared 6-layer Transformer encoder, and decodes them with a shared 6-layer Transformer decoder. The distinctive operator is Multi-View Cross-Attention:

$F^i_{\mathrm{post}}$6

and symmetrically for the other direction. Localization is therefore learned as cross-view region retrieval rather than conventional image-level invariance alone (Chen et al., 2023).

In semi-supervised video object segmentation, SITVOS uses a Siamese ResNet50 backbone for memory frames and the current frame, a transformer encoder over flattened past-frame tokens, a transformer decoder over current-frame tokens, and a Feature Interaction Module that performs mutual cross-attention between the two streams. Memory masks are embedded by a ResNet18 whose first convolution accepts a single channel, and the interaction module uses the mask-filtered memory $F^i_{\mathrm{post}}$7 to suppress background in the decoder-side cross-attention. The output is a current-frame mask obtained by a segmentation decoder with refinement modules and skip connections (Lan et al., 2021).

The same family extends beyond dense prediction. In monocular 3D human pose estimation, the Siamese branches process paired 2D keypoint detections from calibrated cameras, the encoder maps them into a latent $F^i_{\mathrm{post}}$8 of $F^i_{\mathrm{post}}$9 normalized 3D vectors, and the decoder lifts the latent code to 3D joints while a Siamese equivariance loss enforces $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$0 for the known relative rotation $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$1 (Véges et al., 2018). In self-supervised learning, SidAE couples a shared ResNet-18 encoder plus projector with a predictor head and a shared decoder built from a fully connected layer and five transposed convolutions to simultaneously impose view consistency and denoising reconstruction (Baier et al., 2023). In time-series augmentation, DTWSSE trains a Siamese encoder so that latent Euclidean distance approximates DTW and then uses the decoder to map interpolated latent points back to the time-series space, effectively replacing raw-space SMOTE interpolation with DTW-aligned latent interpolation (Yang et al., 2021).

5. Reported empirical behavior

The empirical record is mixed and task-dependent rather than uniformly favorable. In bridge transliteration, the correlational encoder–decoder surpasses the two-stage encoder–decoder in $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$2 language pairs and exceeds the PBSMT baseline in $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$3 pairs, with a particularly large gain for Ma$$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$4Hi: $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$5 versus $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$6 for the two-stage encoder–decoder. By contrast, in bridge captioning the correlational model trails both strong baselines, scoring BLEU-4 $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$7, ROUGE-L $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$8, and CIDEr $$\mathrm{Conv2D}\rightarrow \mathrm{ReLU}\rightarrow \mathrm{BatchNorm}$$9, versus BLEU-4 $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$0, ROUGE-L $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$1, and CIDEr $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$2 for the two-stage pipeline. The model nevertheless produces plausible French captions from images without any Image–French training pairs (Saha et al., 2016).

For remote sensing change detection, ChangeFormer reports on LEVIR-CD test data $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$3, $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$4, and $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$5, compared with BIT at $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$6, $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$7, and $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$8, corresponding to $D_1=\{(x_i,z_i)\}_{i=1}^{N_1}$9 F1, $X$0 IoU, and $X$1 OA over the previous state of the art. On DSIFN-CD, the gains are larger: $X$2, $X$3, and $X$4, versus BIT at $X$5, $X$6, and $X$7, for reported improvements of $X$8 F1, $X$9 IoU, and $f_Z(\cdot;\theta_Z)$00 OA (Bandara et al., 2022).

For self-supervised alignment in serial-section electron microscopy, SEAMLeSS improves chunked Pearson correlation relative to several baselines. On Pinky40-prealigned images, reported CPC means are None $f_Z(\cdot;\theta_Z)$01, FlowNet $f_Z(\cdot;\theta_Z)$02, SPyNet $f_Z(\cdot;\theta_Z)$03, Alembic $f_Z(\cdot;\theta_Z)$04, and SEAMLeSS $f_Z(\cdot;\theta_Z)$05. The paper further reports better behavior in the $f_Z(\cdot;\theta_Z)$06th and $f_Z(\cdot;\theta_Z)$07th percentile tails and emphasizes that Siamese encoding avoids the worst coarse-level failures seen with SPyNet’s raw image pyramids (Mitchell et al., 2019).

In detection pretraining, Siamese DETR consistently improves transfer across DETR variants. On COCO, Vanilla DETR improves from AP $f_Z(\cdot;\theta_Z)$08 from scratch to AP $f_Z(\cdot;\theta_Z)$09, Conditional DETR from $f_Z(\cdot;\theta_Z)$10 to $f_Z(\cdot;\theta_Z)$11, Deformable DETR single-scale from $f_Z(\cdot;\theta_Z)$12 to $f_Z(\cdot;\theta_Z)$13, and Deformable DETR multi-scale from $f_Z(\cdot;\theta_Z)$14 to $f_Z(\cdot;\theta_Z)$15. On PASCAL VOC, Deformable DETR multi-scale improves from AP $f_Z(\cdot;\theta_Z)$16 from scratch to $f_Z(\cdot;\theta_Z)$17, and Conditional DETR improves from $f_Z(\cdot;\theta_Z)$18 to $f_Z(\cdot;\theta_Z)$19. The gains are larger at AP75 than AP50, which the paper interprets as stronger localization priors. The method also reports AR@100 $f_Z(\cdot;\theta_Z)$20, compared with DETReg at $f_Z(\cdot;\theta_Z)$21, UP-DETR at $f_Z(\cdot;\theta_Z)$22, and random initialization at $f_Z(\cdot;\theta_Z)$23 (Chen et al., 2023).

SITVOS reports DAVIS 2017-Val $f_Z(\cdot;\theta_Z)$24, $f_Z(\cdot;\theta_Z)$25, $f_Z(\cdot;\theta_Z)$26, and $f_Z(\cdot;\theta_Z)$27 FPS; DAVIS 2016-Val $f_Z(\cdot;\theta_Z)$28, $f_Z(\cdot;\theta_Z)$29, and $f_Z(\cdot;\theta_Z)$30; and YouTube-VOS Val overall $f_Z(\cdot;\theta_Z)$31, with $f_Z(\cdot;\theta_Z)$32, $f_Z(\cdot;\theta_Z)$33, $f_Z(\cdot;\theta_Z)$34, and $f_Z(\cdot;\theta_Z)$35. The Feature Interaction Module contributes $f_Z(\cdot;\theta_Z)$36 on DAVIS16 and $f_Z(\cdot;\theta_Z)$37 on DAVIS17 relative to the ablated model without FIM (Lan et al., 2021).

For camera-robust 3D lifting, the equivariant Siamese model reports a Protocol #3 cross-camera average MPJPE of $f_Z(\cdot;\theta_Z)$38 mm on Human3.6M when synthetic cameras are used every $f_Z(\cdot;\theta_Z)$39, outperforming PoseGrammar with the same augmentation at $f_Z(\cdot;\theta_Z)$40 mm. On Protocol #1, the gain is smaller, $f_Z(\cdot;\theta_Z)$41 mm versus $f_Z(\cdot;\theta_Z)$42 mm, consistent with the observation that all training viewpoints are already present. Ablations show that removing the Siamese loss degrades Protocol #3 performance to $f_Z(\cdot;\theta_Z)$43 mm, and removing rotation augmentation degrades it to $f_Z(\cdot;\theta_Z)$44 mm (Véges et al., 2018).

In self-supervised image representation learning, SidAE consistently exceeds both SimSiam and the denoising autoencoder baseline. On CIFAR-10 with $f_Z(\cdot;\theta_Z)$45 labels after $f_Z(\cdot;\theta_Z)$46 pretraining epochs and a frozen encoder, SidAE reaches $f_Z(\cdot;\theta_Z)$47 accuracy, compared with SimSiam at $f_Z(\cdot;\theta_Z)$48 and AE at $f_Z(\cdot;\theta_Z)$49. On Fashion-MNIST with $f_Z(\cdot;\theta_Z)$50 labels, the corresponding figures are $f_Z(\cdot;\theta_Z)$51, $f_Z(\cdot;\theta_Z)$52, and $f_Z(\cdot;\theta_Z)$53; on MNIST they are $f_Z(\cdot;\theta_Z)$54, $f_Z(\cdot;\theta_Z)$55, and $f_Z(\cdot;\theta_Z)$56. On STL-10 pretraining transferred to CIFAR-10, SidAE reports $f_Z(\cdot;\theta_Z)$57 versus $f_Z(\cdot;\theta_Z)$58 for SimSiam and $f_Z(\cdot;\theta_Z)$59 for AE (Baier et al., 2023).

In time-series augmentation, DTWSSE generally outperforms SMOTE and related ablations. On DistalPhalanxTW with a fully connected autoencoder, Top-1 accuracy improves from $f_Z(\cdot;\theta_Z)$60 without augmentation and $f_Z(\cdot;\theta_Z)$61 with SMOTE to $f_Z(\cdot;\theta_Z)$62 with DTWSSE; Worst accuracy improves from $f_Z(\cdot;\theta_Z)$63 and $f_Z(\cdot;\theta_Z)$64 to $f_Z(\cdot;\theta_Z)$65; and Average accuracy improves from $f_Z(\cdot;\theta_Z)$66 and $f_Z(\cdot;\theta_Z)$67 to $f_Z(\cdot;\theta_Z)$68. On balanced datasets, DTWSSE(CNN) reaches Top-1 $f_Z(\cdot;\theta_Z)$69 and Average $f_Z(\cdot;\theta_Z)$70 on Chinatown, and Top-1 $f_Z(\cdot;\theta_Z)$71 and Average $f_Z(\cdot;\theta_Z)$72 on MelbournePedestrian. On Libras, DTWSSE(DNN) reports Average $f_Z(\cdot;\theta_Z)$73, compared with $f_Z(\cdot;\theta_Z)$74 for SMOTE and $f_Z(\cdot;\theta_Z)$75 without augmentation (Yang et al., 2021).

6. Limitations, misconceptions, and extension points

A frequent misconception is that any Siamese encoder–decoder must share encoder weights. The correlational encoder–decoder contradicts that assumption directly: its encoders are separate networks with $f_Z(\cdot;\theta_Z)$76, and the shared structure is enforced in representation space by correlation rather than in parameter space. A second misconception is that the decoder must be generative in the narrow sense of text or image reconstruction. In this literature, the decoder may instead be an MLP segmentation head, a hierarchical aligner, a DETR decoder with cross-view attention, or a 3D pose regressor (Saha et al., 2016, Bandara et al., 2022, Mitchell et al., 2019, Chen et al., 2023, Véges et al., 2018).

The limitations are equally domain-specific. The correlational encoder–decoder is sensitive to pivot quality and domain, requires tuning of $f_Z(\cdot;\theta_Z)$77, and lags strong baselines on captioning; the paper notes that attention and richer conditioning may be needed for complex multimodal semantics. ChangeFormer assumes co-registered input pairs and can degrade under misregistration, while still facing transformer memory–computation trade-offs despite sequence reduction. SEAMLeSS requires smoothness regularization for self-supervised training and needs masking near discontinuities such as cracks and folds to avoid oversmoothing true non-smooth transforms (Saha et al., 2016, Bandara et al., 2022, Mitchell et al., 2019).

Further limitations arise from the way paired correspondence is constructed. Siamese DETR depends on view overlap and the quality of object proposals, although it is reported to remain robust across proposal sources and to benefit from EdgeBoxes. SITVOS remains challenged by severe background distractors that closely resemble the target and by extremely fast motion, even though spatio-temporal aggregation and mask gating improve robustness to occlusion, deformation, and disappearance. The pose-estimation model depends on calibrated relative camera rotations and still exhibits failure modes in sitting poses, which the paper attributes to dataset bias toward standing (Chen et al., 2023, Lan et al., 2021, Véges et al., 2018).

Self-supervised and augmentation-oriented E2E-R models introduce another set of trade-offs. SidAE requires tuning the interpolation weight $f_Z(\cdot;\theta_Z)$78: large $f_Z(\cdot;\theta_Z)$79 moves the model toward pixel-level reconstruction and can weaken invariance, while the empirical results indicate that lower $f_Z(\cdot;\theta_Z)$80 values are often preferable on harder datasets such as CIFAR-10 and STL-10. DTWSSE inherits the $f_Z(\cdot;\theta_Z)$81 cost of DTW for neighbor selection, trains encoder and decoder sequentially rather than jointly, and shows dataset dependence in the choice between CNN and fully connected encoder–decoder variants. The paper also reports that a naive autoencoder without DTW-aligned latent geometry can sharply degrade worst-case accuracy on some datasets (Baier et al., 2023, Yang et al., 2021).

The literature also suggests several extension points. The pivot-generation work explicitly proposes attention, contrastive correlations, or Deep CCA as ways to strengthen shared-space learning while preserving end-to-end decodability. ChangeFormer’s analysis points to domain adaptation under sensor and seasonal shift. Siamese DETR discusses extensions to segmentation, panoptic detection, video detection, and unified end-to-end pretraining of both backbone and Transformer. DTWSSE notes that warping windows, lower bounds, joint objectives, or additional latent regularization could reduce cost or improve stability. These proposals reinforce a general conclusion: E2E-R is not a fixed recipe but a reusable architectural schema in which paired encoders and a decoder are optimized together so that inter-branch comparability directly supports the downstream prediction task (Saha et al., 2016, Bandara et al., 2022, Chen et al., 2023, Yang et al., 2021).