Hybrid Conv3D+ConvLSTM for Spatiotemporal Data

• Hybrid Conv3D+ConvLSTM architectures are neural models combining 3D convolutions for local spatiotemporal feature extraction with ConvLSTM units for temporal integration.
• They are effectively applied in tasks like silent speech regression and semantic segmentation, balancing detailed local descriptors with global context via SE attention.
• Ablation studies highlight that a configuration of three Conv3D layers followed by a single ConvLSTM layer optimizes accuracy while reducing computational cost.

Hybrid Conv3D + ConvLSTM architectures are neural network models that combine three-dimensional convolutions (Conv3D) with convolutional long short-term memory (ConvLSTM) blocks, enabling efficient end-to-end learning from spatiotemporal data. These hybrids are designed to exploit the strengths of both operations: Conv3D for early local spatiotemporal pattern extraction and ConvLSTM for temporal integration and memory, while often incorporating channel-wise attention mechanisms such as squeeze-and-excitation (SE) for enhanced global context. Hybrid designs have demonstrated improved accuracy, computational efficiency, and parameter economy across diverse domains, including silent speech interfaces and dense semantic segmentation.

1. Mathematical Foundations

Conv3D applies kernels over both spatial and temporal dimensions to jointly capture local motion and appearance cues within short fixed-length input windows. For an input $X \in \mathbb{R}^{T \times H \times W \times C_i}$, kernel $$W \in \mathbb{R}^{d_t \times d_h \times d_w \times C_i \times C_o}$$, and output $$Y \in \mathbb{R}^{T' \times H' \times W' \times C_o}$$, the convolution is defined as:

$$Y(t,x,y,k) = \sum_{c=0}^{C_i-1} \sum_{i=0}^{d_t-1} \sum_{j=0}^{d_h-1} \sum_{l=0}^{d_w-1} W(i,j,l,c,k)\cdot X(t\cdot s_t + i, x\cdot s_h + j, y\cdot s_w + l, c)$$

where $s_t,s_h,s_w$ denote temporal and spatial strides.

ConvLSTM extends standard LSTM memory to structured spatial data by replacing matrix multiplications by convolutions in all gates and updating hidden and cell states at each spatial location: $$\begin{aligned} f_t &= \sigma(W_{xf} * X_t + W_{hf} * H_{t-1} + W_{cf} \circ C_{t-1} + b_f) \ i_t &= \sigma(W_{xi} * X_t + W_{hi} * H_{t-1} + W_{ci} \circ C_{t-1} + b_i) \ \tilde{C}_t &= \tanh(W_{xc} * X_t + W_{hc} * H_{t-1} + b_c) \ C_t &= f_t \circ C_{t-1} + i_t \circ \tilde{C}_t \ o_t &= \sigma(W_{xo} * X_t + W_{ho} * H_{t-1} + W_{co} \circ C_t + b_o) \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$ Here, $*$ denotes convolution, $\circ$ the Hadamard product, and all weights are convolutional kernels.

2. Architectures and Layer Coupling Strategies

Hybrid Conv3D + ConvLSTM systems can be instantiated in multiple forms, with coupling determined by domain and task requirements.

Silent Speech Regression (Ultrasound-to-Spectrogram)

"Improved Processing of Ultrasound Tongue Videos by Combining ConvLSTM and 3D Convolutional Networks" presents the following pipeline (Shandiz et al., 2022):

• Input: 25 ultrasound frames (128 × 64), single-channel.
• Backbone: Three Conv3D layers for spatiotemporal feature extraction, each followed by dropout (p=0.35), aggressive spatial downsampling, and temporal stride only in the first Conv3D (stride 5).
• ConvLSTM2D: Single layer (filters=64, kernel=3×3, strides=2) for global temporal integration over the five remaining feature sequences.
• Dense output: Flatten → Dense(80, linear) for mel-spectrogram regression.

Layer-by-layer summary:

Layer	Type	Configuration
1	Conv3D	30 filters, (5x13x13), stride (5,2,2)
+Dropout		p = 0.35
2	Conv3D	60 filters, (1x13x13), stride (1,2,2)
+Dropout		p = 0.35
+MaxPool3D		(1,2,1)
3	Conv3D	90 filters, (1x13x13), stride (1,2,2)
+Dropout		p = 0.35
4	ConvLSTM2D	64 filters, (3x3), stride (2,2)
5	Flatten
6	Dense	80 units, linear activation

Semantic Segmentation (REthinker Blocks)

"RethNet: Object-by-Object Learning for Detecting Facial Skin Problems" utilizes a REthinker block composed of local Conv3D or ConvLSTM units and a squeeze-and-excitation (SE) branch (Bekmirzaev et al., 2021):

• Input: Feature map $U_d \in \mathbb{R}^{H \times W \times D}$.
• Patchification: Spatial division into $N^2$ blocks ($N=4$), each block treated as a time step for convLSTM or Conv3D.
• Spatiotemporal module: Either Conv3D (kernel 3x3x3, stride 1) or ConvLSTM (kernel 3x3, stride 1, unrolled over patches), maintaining spatial and channel correspondence.
• SE channel attention: Global spatial pooling, two dense layers (reduction ratio $r=16$), sigmoid gating, and channelwise scaling of output.
• Output: Reassembled feature map with SE scaling and possible skip connection.

3. Training Protocols and Hyperparameters

Silent Speech Regression

• Dataset: 438 utterances (~0.5 hr) from one speaker, split 310/41/87 (train/dev/test).
• Preprocessing: Frames downsampled to 128 × 64, normalized to [–1, 1]; target mel-spectrogram, 80 bands, z-scored.
• Loss: Mean Squared Error over 80 outputs.
• Optimizer: Adam, learning rate 0.001, default $\beta_1=0.9$, $\beta_2=0.999$.
• Batch size: 32, trained for 50–80 epochs (until dev MSE plateaus at ≈ 0.28).

REthinker Block for Segmentation

• Optimizer: SGD with momentum (μ=0.9), initial learning rate $1\times 10^{-3}$, decay every 50 epochs, 200 total epochs, weight decay $10^{-4}$.
• Loss: Pixelwise softmax cross-entropy over 17 classes.
• Augmentation: Random rotation (±30°), zoom (0.8–1.2), flip.
• Input: 512 × 512 crops.

4. Empirical Performance and Ablation

Quantitative Comparison (Ultrasound-to-Speech)

Model	Dev MSE	Dev R²	Test MSE	Test R²
3D-CNN (baseline)	0.292	0.714	0.293	0.710
3D-CNN + BiLSTM	0.285	0.721	0.282	0.721
3D-CNN + ConvLSTM (hybrid)	0.276	0.727	0.276	0.730

• Ablation: Pure ConvLSTM (3–4 stacked) increases MSE and slows training (1.8× baseline time). The Conv3D×3 + ConvLSTM hybrid delivers the lowest MSE and fastest epoch time (0.9× baseline).

Quantitative Comparison (REthinker Block)

Model	mIoU (%)
Baseline Deeplab v3+	64.12
+SE	65.49
+Patch Conv	65.52
+Conv3D+SE (R-d)	76.56
+ConvLSTM+SE (R-e)	79.46

• Interpretation: The largest accuracy improvement is from adding full spatiotemporal Conv3D+SE, with ConvLSTM+SE yielding the highest result, attributed to explicit modeling of long-range co-occurrences among object patches.

5. Mechanisms and Modeling Rationale

Hybrid Conv3D + ConvLSTM modules excel by partitioning the workload between:

• Conv3D: Encodes local (short-term) spatiotemporal context and aggressively reduces spatial resolution, extracting robust low-level descriptors and short-range motion patterns efficiently.
• ConvLSTM: Integrates memory and sequential dependencies along time (ultrasound tongue video) or across patch sequences (spatially “flattened time” in segmentation REthinkers), capturing long-range structure, temporal continuity, and complex co-occurrence relations.
• SE attention: Provides global context and dynamic channel-wise feature recalibration, substantially improving semantic segmentation where subtle inter-class relations matter.

Ablation studies confirm that stacking more ConvLSTM layers or interleaving Conv3D and ConvLSTM does not yield further gains; the optimal design is three Conv3Ds (for local features) followed by a single ConvLSTM (for global fusion) (Shandiz et al., 2022).

6. Applications and Broader Impacts

Hybrid Conv3D + ConvLSTM architectures have demonstrated state-of-the-art performance in:

• Silent speech interface regression: Mapping ultrasound tongue video to mel-spectrogram vectors with higher accuracy, reduced depth, and lower computation than pure 3D-CNN or multi-layer ConvLSTM approaches (Shandiz et al., 2022).
• Medical image segmentation: In RethNet’s semantic segmentation of facial lesions, REthinker blocks leveraging Conv3D/ConvLSTM with SE boost both local discrimination and global co-occurrence modeling, achieving mean intersection-over-union scores significantly above baselines (Bekmirzaev et al., 2021).

This architecture class is well-suited for any task requiring efficient joint modeling of fine-grained spatial features and extended temporal/contextual dependencies, such as video-to-text, dynamic MR/CT analysis, or fine-grained action recognition.

7. Limitations and Considerations

While Conv3D + ConvLSTM hybrids offer superior accuracy and efficiency, their optimality is contingent on task-specific data structure:

• Tasks with weak temporal dependencies or highly redundant spatial patterns may see diminishing returns from ConvLSTM integration.
• Architectural tuning (e.g., number of Conv3D layers, use of SE, patch grid choices) remains essential.
• Training speed-ups are realized in part due to shallower depth compared to stacked ConvLSTM alternatives; however, ConvLSTM units are more parameter-intensive than single Conv3D layers.

A plausible implication is that future extensions may focus on adaptive gating between Conv3D and ConvLSTM modules, or integration with transformer-based attention for even richer spatiotemporal modeling.