Spatio-Temporal Fusion Model for Standard View Classification of Echocardiographic Videos

Abstract: Automated classification of standard echocardiographic views is crucial for efficient clinical workflow but faces three main challenges. First, publicly available datasets are scarce and limited in scale and view coverage. Second, the performance of some modern video-level architectures for echocardiographic view classification remains underexplored. Third, some view categories exhibit highly similar spatial appearances, making single-frame features insufficient for discrimination, while heterogeneous frame quality complicates robust temporal information fusion. To address these challenges, we release the Echocardiographic Videos of Nine Views (EV9V) dataset, comprising 5,138 videos, 910,579 frames, and 9 standard views, which is, to the best of our knowledge, the largest publicly available echocardiography video dataset. Using EV9V, we systematically benchmark representative video classification architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers. Furthermore, we propose a Spatio-Temporal Fusion Model (STFM), an efficient dual-stream CNN-LSTM (Long Short-Term Memory) framework that jointly captures spatial anatomical structures and temporal cardiac dynamics. The proposed framework leverages uncertainty-aware learning to preferentially sample representative video segments during training and evidence-based fusion during inference, improving robustness to variations in frame quality across echocardiographic videos. Extensive experiments demonstrate that our method achieves competitive performance across diverse video classification models, validating the effectiveness of uncertainty-aware spatio-temporal learning for echocardiographic view classification. The code is available at https://github.com/bgx666/stfm.

• The paper introduces the EV9V dataset and the Spatio-Temporal Fusion Model (STFM) that fuses spatial and temporal features for accurate echocardiographic view classification.
• The paper leverages a dual-stream CNN–LSTM architecture with uncertainty-guided selective sampling to enhance prediction accuracy while reducing computational complexity.
• The paper demonstrates state-of-the-art performance with 94.48% accuracy and 91.14 macro-F1, outperforming larger models and offering interpretable uncertainty for clinical applications.

Spatio-Temporal Fusion Model for Standard View Classification of Echocardiographic Videos

Introduction and Problem Motivation

Automated echocardiographic view classification is essential for downstream computer-aided cardiac assessment, but prior work has been hampered by limited dataset availability, inconsistent benchmarks, and the technical challenge of discriminating views with highly similar spatial anatomy using suboptimal-quality clinical videos. Most accessible datasets are either image-based or private, impeding reproducibility. Additionally, recent advances in video modeling—e.g., transformers and compound CNNs—had not been systematically explored for echocardiography. The paper introduces the EV9V dataset and the Spatio-Temporal Fusion Model (STFM) to address these deficiencies (2606.17437).

The EV9V Dataset

EV9V is introduced as the largest publicly available benchmark for video-based echocardiographic view classification, comprising 5,138 videos (910,579 frames) across nine clinically standard views with rigorous three-stage expert annotation and real-world acquisition heterogeneity. The dataset's multi-view coverage includes complex and underrepresented views such as PMPALA, critical for comprehensive echocardiographic interpretation. Partitioning is performed at the patient level to prevent data leakage, and the dataset exposes natural class imbalance and variable sequence length, mirroring practical deployment settings.

Figure 1: Illustration of the nine standard transthoracic echocardiographic views in the EV9V dataset.

A key statistical characterization is the class and frame length distribution across splits, indicating substantial variance in video duration and sample counts per view.

Figure 2: Class distribution across the nine echocardiographic views in the training, validation, and test sets.

Spatio-Temporal Fusion Model (STFM)

STFM is a dual-stream CNN–LSTM architecture balancing compactness and discriminative power. It disentangles spatial and temporal representation learning: an anchor frame is used for spatial modeling, while a temporally centered clip captures the motion patterns unique to each view. Both branches share a shallow CNN stem to minimize redundancy; the temporal stream processes local clips using a lightweight CNN head and multi-layer LSTM.

After spatial and temporal embeddings are concatenated, a compact MLP produces a fused latent, which feeds into a Reliable Evidential Deep Learning (Re-EDL) head, parameterizing a Dirichlet distribution over classes for evidential uncertainty quantification.

Figure 3: Overview of the proposed Spatio-Temporal Fusion Model (STFM).

This design trades off the computational burden of 3D CNNs and the instability of optical-flow two-stream methods by leveraging uncertainty-aware temporal sampling and evidence-weighted aggregation.

Uncertainty-Guided Training and Inference

A critical methodological advancement is the use of uncertainty-guided selective sampling. Rather than uniformly sampling temporal segments for training, a global segment bank maintains exponentially-weighted certainty estimates (driven by Re-EDL uncertainty). An $\epsilon$-greedy policy prioritizes high-certainty segments, with stochastic exploration to avoid mode collapse. This sampling mechanism maximizes the model's focus on representative, high-evidence subsequences.

At inference, multiple uniformly sampled clips are processed, and their evidential Dirichlet parameters are fused, accentuating the contribution of frames with strong, confident evidence. Ablations demonstrate that selective sampling significantly improves uncertainty calibration and test classification performance relative to random sampling.

Benchmarking and Empirical Results

A comprehensive evaluation compares 2D CNNs, temporal aggregation backbones, RNN-based models, 3D CNNs, and video transformers using unified protocols on EV9V. RegNet achieves the highest frame-level accuracy among 2D methods (93.32% test accuracy, 89.02 macro-F1), but all top-performing models fail to distinguish challenging anatomical neighbors due to single-frame ambiguity.

For video-level classification, STFM with ResNet-18 backbone sets a new state of the art: 94.48% accuracy and 91.14% macro-F1 on the test split, outperforming UniFormerV2 (the best video transformer) with 114.25M parameters versus STFM's 14.75M. Ablation experiments confirm that increased backbone capacity offers diminishing returns, temporal component depth can be minimal without affecting performance, and the method is robust to Re-EDL hyperparameters.

Figure 4: Per-class test recall (%) comparison between RegNet, UniFormerV2, and STFM, highlighting recall gains in challenging views.

Interpretability analyses reveal that improvements are most pronounced for pairs of views characterized primarily by temporal motion cues. For difficult confusion pairs, such as PMASA ↔ PMPALA and A5C ↔ A4C, STFM consistently outperforms single-frame and transformer methods.

Figure 5: Representative examples from the five most frequent confusion pairs on the EV9V test set.

Uncertainty Quantification and Selective Sampling

The impact of selective sampling on evidential uncertainty is explicit: segment-level uncertainty trajectories show wider dynamic range and better outlier detection under the selective regime ($\epsilon = 0.2$), supported by increased per-video uncertainty range and variance in population-level statistics.

Figure 6: Segment-level uncertainty trajectories for a representative PMASA test video, comparing selective ($\epsilon=0.2$) and random ($\epsilon=1.0$) sampling strategies.

Implications and Future Directions

The public release of EV9V and the STFM approach alters the landscape of echocardiography AI research. STFM demonstrates that compact spatio-temporal architectures, coupled with uncertainty-based training curricula, are highly effective for clinical video data, mitigating the need for prohibitively large video transformers and 3D CNNs. The evidence-based aggregation not only aids robust prediction but also provides interpretable uncertainty profiles for clinical integration, supporting downstream tasks such as quality control and out-of-distribution detection.

Theoretically, this work substantiates the value of incorporating segment-level evidential modality and curriculum-based sampling within medical video learning paradigms. Practically, the model's efficiency and generalization capabilities are conducive to clinical deployment where computational budgets and annotation reliability are key constraints.

Immediate future work includes scaling STFM with self-supervised pretraining on EV9V, extending to multi-label multi-view scenarios, and exploring ensemble and semi-supervised settings leveraging the uncertainty estimates for active annotation strategies. Furthermore, non-convolutional spatial backbones and more sophisticated temporal attention mechanisms present promising avenues for further gains without significantly increasing computational cost.

Conclusion

This research provides strong empirical and methodological evidence that spatio-temporal fusion with uncertainty-guided sampling is optimal for standard view classification in echocardiographic video. The EV9V dataset sets a new bar for robust, reproducible model development in this space, and the STFM framework offers a practical solution balancing accuracy, interpretability, and efficiency for real-world clinical application (2606.17437).