End2EndNet: Joint Optimization in Vision & Emotion

• End2EndNet is a unified approach that jointly trains multiple submodels to directly estimate fundamental matrices from stereo images and map joint-coordinate sequences to emotion labels.
• It replaces classical multi-stage pipelines by integrating correspondence finding, outlier rejection, and model fitting (or spatio-temporal encoding) into a single trainable framework.
• The architecture leverages geometry-aware losses and attention mechanisms to enforce epipolar constraints in vision and to optimize deep representations in skeleton-based emotion recognition.

End2EndNet is a term used in two distinct but conceptually aligned senses in recent literature. In computer vision, it denotes a specific differentiable architecture for estimating the fundamental matrix directly from stereo images by jointly performing correspondence finding, outlier rejection, and matrix estimation (Zhang et al., 2020). In affective computing, it denotes a taxonomy category for skeleton-based emotion recognition in which raw joint-coordinate sequences are mapped directly to emotion labels by a unified deep model without handcrafted feature extraction (Lu et al., 24 Jul 2025). In both usages, the defining characteristic is joint optimization across stages that are often separated in classical pipelines.

1. Terminological scope and core meaning

The two uses of the term share an architectural principle rather than a common task. In the fundamental-matrix setting, End2EndNet is a named model. In the skeleton-emotion setting, End2EndNet is a paradigm within a four-fold taxonomy: Traditional, Feat2Net, FeatFusionNet, and End2EndNet (Lu et al., 24 Jul 2025).

Usage	Input	Output
Fundamental-matrix estimation	Stereo pair $I_1, I_2 \in \mathbb{R}^{H \times W \times 3}$	Normalized $3 \times 3$ fundamental matrix
Skeleton-based emotion recognition	Skeleton sequence of $T$ frames, $N$ joints, and $D$ coordinates	Emotion label probabilities

The common meaning of “end-to-end” is therefore not a single network topology, but the removal of manually separated processing stages in favor of a single trainable pipeline. A recurrent misconception is to treat End2EndNet as a single canonical architecture across domains. The available sources instead support two domain-specific usages: one concrete and geometry-constrained, the other categorical and task-agnostic within skeleton-based emotion recognition.

2. End2EndNet for fundamental-matrix estimation

For fundamental-matrix estimation, End2EndNet is organized as three jointly trained submodels that mirror the classic pipeline of feature matching, outlier rejection, and model fitting (Zhang et al., 2020). The first submodel, the putative correspondence finder, takes the stereo pair as input and uses a UCN-style Siamese CNN with shared weights, no spatial transformers, location-aware pooling layers, and additional convolutions. Its output is a set of $K$ putative correspondences

$\{(m_i,m'_i)\}_{i=1\ldots K},$

where $m_i=(x_i,y_i)$ and $m'_i=(x'_i,y'_i)$, arranged as a feature tensor of shape $K \times 4$.

The second submodel performs outlier rejection. It consumes the $3 \times 3$0 correspondence tensor and applies six PointCN blocks, described as PointNet with Context-Norm, together with ResNet-style MLPs applied per correspondence and a shared MLP head. The output is a scalar weight

$3 \times 3$1

for each putative match, yielding a tensor of shape $3 \times 3$2. These weights represent the estimated inlier probability.

The third submodel estimates the fundamental matrix from the concatenated correspondence-plus-weight tensor of shape $3 \times 3$3. It uses fully connected layers and an “F-matrix Reconstruction Layer” that outputs $3 \times 3$4 and two scalars $3 \times 3$5, then forms

$3 \times 3$6

which guarantees rank $3 \times 3$7. The resulting matrix is normalized as

$3 \times 3$8

The geometric object being estimated is the fundamental matrix $3 \times 3$9 with $T$0, which encodes the epipolar constraint

$T$1

for corresponding homogeneous points $T$2 and $T$3. The corresponding epipolar line in the second image is

$T$4

The summary also gives the symmetric epipolar distance

$T$5

Because all three submodels are differentiable, the full pipeline is trainable end to end. A plausible implication is that the architecture attempts to absorb the statistical role of classical multi-stage geometry pipelines while retaining explicit algebraic constraints on $T$6.

3. Geometry-preserving supervision and epipolar-angle evaluation

The training objective for the fundamental-matrix model is a composite loss

$T$7

designed to preserve geometry while also regressing matrix entries (Zhang et al., 2020). The entry-wise component with respect to ground truth $T$8 is

$T$9

The epipolar-constraint term is computed over $N$0 inlier matches, selected with respect to $N$1 by symmetric distance $N$2:

$N$3

The reported hyperparameters are

$N$4

This loss addresses a limitation of pure entry-wise regression: numerical similarity between matrix entries does not by itself guarantee satisfactory epipolar geometry. The summary explicitly states that the loss both pulls $N$5 toward ground truth in entry space and enforces the core epipolar constraint.

The evaluation protocol introduces a new metric, the Inlier Epipolar Angle $N$6. The stated motivation is that common metrics such as mean epipolar error $N$7 and symmetric epipolar distance $N$8 measure point-to-line distance but can fail to detect orientation errors in the epipolar “fan.” For each inlier correspondence $N$9, the method compares the ground-truth line

$D$0

and the estimated line

$D$1

If the estimated line still passes through $D$2 within the inlier threshold, the acute angle $D$3 between the two lines is measured, and

$D$4

The importance of $D$5 is methodological as much as numerical. It formalizes the paper’s claim that visually plausible epipolar fans require correct line orientation, not merely low residual distance. This is one of the clearest places where “end-to-end” does not mean unconstrained black-box learning: the architecture is paired with geometry-aware supervision and geometry-aware evaluation.

4. Datasets, quantitative results, and ablations for the geometry model

The fundamental-matrix model is evaluated on KITTI and TUM SLAM, each with 2,000 images split into 1,600/200/200 for train/validation/test; ground-truth $D$6 is computed from known intrinsics and extrinsics, and matches are pre-filtered by symmetric epipolar distance $D$7 (Zhang et al., 2020). The three submodels are trained jointly end to end. The summary reports no special data augmentations beyond standard random cropping and color jitter, and states that the network converges in the order of tens of epochs, with the exact schedule not specified.

On 200 held-out test image pairs from each dataset, the reported results are as follows.

Method	KITTI: $D$8	TUM SLAM: $D$9
SIFT+8Point	5.53 / $K$0 / $K$1	10.88 / $K$2 / $K$3
SIFT+RANSAC	0.42 / 10.88 / $K$4	0.99 / 16.08 / $K$5
DeepF	0.62 / 118.66 / $K$6	1.30 / 20.32 / $K$7
SIFT+OANet+RANSAC	0.27 / 43.48 / $K$8	0.61 / 9.83 / $K$9
Ours	0.043 / 0.72 / $\{(m_i,m'_i)\}_{i=1\ldots K},$0	0.81 / 6.57 / $\{(m_i,m'_i)\}_{i=1\ldots K},$1
GroundTruth	0.0084 / 0.63 / $\{(m_i,m'_i)\}_{i=1\ldots K},$2	0.045 / 0.035 / $\{(m_i,m'_i)\}_{i=1\ldots K},$3

The summary states that the model achieves “the best trade-off of point–line distance and line orientation” and comes very close to the GroundTruth fan on KITTI. On TUM SLAM, the reported behavior is less uniformly close to GroundTruth, but the method still improves substantially over several baselines in $\{(m_i,m'_i)\}_{i=1\ldots K},$4 and $\{(m_i,m'_i)\}_{i=1\ldots K},$5.

The ablation study isolates the effect of the epipolar-constraint term. On KITTI, the reported “Ours w/o $\{(m_i,m'_i)\}_{i=1\ldots K},$6” result is

$\{(m_i,m'_i)\}_{i=1\ldots K},$7

compared with DeepF at $\{(m_i,m'_i)\}_{i=1\ldots K},$8 and OANet+RANSAC at $\{(m_i,m'_i)\}_{i=1\ldots K},$9. This shows that even without $m_i=(x_i,y_i)$0, the architecture yields low angular error and improved symmetric distance relative to prior one-stage networks, while point-line error remains worse. Adding $m_i=(x_i,y_i)$1 changes DeepF to $m_i=(x_i,y_i)$2 and changes the proposed model from $m_i=(x_i,y_i)$3 to $m_i=(x_i,y_i)$4. The summary attributes these changes to the geometry-preserving loss, particularly for entry-wise accuracy and fan orientation.

5. End2EndNet as a taxonomy category in skeleton-based emotion recognition

In the 2025 survey on emotion recognition from skeleton data, End2EndNet is not a single named network but the final category in a unified taxonomy of methods (Lu et al., 24 Jul 2025). It is defined as a class of models that ingest raw skeleton sequences, consisting of joint 3D coordinates over time, and jointly learn spatio-temporal feature representations together with the final emotion prediction in one unified pipeline. No handcrafted feature extraction is required.

The stated motivation has three parts. First, End2EndNet eliminates manual feature engineering and its associated bias. Second, it uses deep architectures such as RNNs, CNNs, GCNs, and Transformers to discover discriminative features directly. Third, it can improve accuracy by optimizing all parameters jointly under a single loss function.

The survey summarizes a prototypical architecture with four major stages after input formation. The input is a skeleton sequence of shape $m_i=(x_i,y_i)$5, where $m_i=(x_i,y_i)$6 or $m_i=(x_i,y_i)$7, optionally followed by normalization such as root-joint centering or length scaling. A spatio-temporal encoder then processes the sequence. The survey lists several encoder families: LSTM or GRU stacks, often with temporal attention; CNN variants that encode sequences as pseudo-images; GCN variants that define a graph $m_i=(x_i,y_i)$8 over joints and apply spatial-temporal graph convolutions; and Transformer variants that use self-attention over time or graph nodes. Temporal pooling or sequence aggregation follows, typically as global average pooling over time or attention-weighted pooling. One or more fully connected layers with Softmax then produce emotion label probabilities.

The mathematical building blocks given in the survey include a graph-convolutional layer

$m_i=(x_i,y_i)$9

where $m'_i=(x'_i,y'_i)$0 is the normalized adjacency, possibly learnable, $m'_i=(x'_i,y'_i)$1 are learnable weights, and $m'_i=(x'_i,y'_i)$2 is a nonlinearity such as ReLU. For attention-based variants, the survey gives

$m'_i=(x'_i,y'_i)$3

The standard classification loss is cross-entropy,

$m'_i=(x'_i,y'_i)$4

and some methods use a combined objective

$m'_i=(x'_i,y'_i)$5

when graph-structure learning is involved. The survey further notes that some methods add weight decay with AdamW or auxiliary losses such as reconstruction in autoencoders.

Representative encoder families listed in the survey include Sapiński et al. 2019 for LSTM-based modeling, Zhang et al.’s AS-LSTM 2021 for temporal attention, Shi et al.’s ST-GCN, Ghaleb et al.’s ST-GCN with attention, Shirian et al.’s L-GrIN, and GaitCycFormer for gait-based Transformer modeling.

6. Relation to adjacent paradigms, benchmark profile, and limitations

The survey distinguishes End2EndNet from the two preceding neural paradigms, Feat2Net and FeatFusionNet (Lu et al., 24 Jul 2025). In Feat2Net, handcrafted features such as joint angles and velocities are computed offline and only the classifier is learned. In FeatFusionNet, handcrafted features are injected into the network through concatenation or fusion losses, so the system is jointly trained but still depends on manual feature design. End2EndNet differs by using raw coordinates as the only input and learning both spatial and temporal representations in a fully data-driven manner. The survey characterizes this as conceptually simpler from the standpoint of preprocessing, while also noting that it generally requires deeper or more complex architectures.

The benchmark excerpts included in the survey show a mixed empirical profile. On the EGBM dataset with 7 emotions and 560 samples, Sapinski et al. (LSTM) achieve 69.00% and AS-LSTM achieves 74.00%, while Wang et al. 2025 and Wang et al. 2023, both marked as manual+FC, achieve 97.43% and 95.55%, respectively. On the KDAE dataset with 7 emotions and 1,402 samples, Ghaleb et al. (ST-GCN) achieve 65.00%, Wang et al. 2025 (manual+FC) achieve 96.67%, and Oğuz et al. (manual+NN) are listed at 99.99% with hold-out evaluation. On the Emilya dataset with 8 emotions and 8,206 samples, Beyan et al. (dual-branch CNN) achieve 96.59%, while Wang et al. 2023 (manual+FC) achieve 94.42%. The survey’s synthesized conclusion is that among pure End2EndNet models based on LSTM, CNN, or GCN, accuracies range from the mid-60s to the mid-70s, and rise above 80–90% only when strong attention, multiscale, or hybrid fusion techniques are applied.

The advantages and limitations are presented in equally direct terms. The listed advantages are fully automated feature learning with minimal human bias, architectural flexibility for incorporating advances such as self-attention, transformers, and generative pretraining, and the possibility that a single end-to-end pipeline is easier to maintain and optimize than a multi-stage system. The listed limitations are data hunger on small datasets, reduced interpretability because there are no explicit motion cues to inspect, and higher computational cost for graph and Transformer models.

The survey also outlines several improvement directions: pre-training or self-supervised learning on large unlabeled skeleton corpora such as MPOSE2021 before fine-tuning on emotion labels; incorporation of small sets of handcrafted priors, including motion energy and joint covariance, as auxiliary tasks or regularizers; lightweight architectures such as pruned GCNs and distilled Transformers for real-time on-edge deployment; and extension to multimodal or multi-person frameworks for improved robustness in the wild. These suggestions clarify another common misconception: “end-to-end” in this literature does not imply the exclusion of priors, auxiliary objectives, or structure regularization. Rather, it denotes that raw-to-prediction mapping is learned within a single optimization framework.