Separable 3D Convolutions (S3D)

• Separable 3D convolutions are defined as design patterns that decompose standard 3D convolutions into spatial, temporal, and channel-wise operations for improved efficiency.
• They reduce computational load by factorizing 3D filters into lower-dimensional kernels, achieving significant reductions in parameters and FLOPs while maintaining accuracy.
• S3D modules integrate seamlessly into architectures like Inception-I3D and ResNet, enabling effective 2D to 3D weight transfer and better speed-to-accuracy trade-offs.

Separable 3D Convolutions (S3D) refer to a family of architectural modules and design patterns for efficiently parameterizing three-dimensional convolutional neural networks. By factorizing the standard 3D convolutional operation into separable components—such as spatial and temporal kernels, depthwise and pointwise kernels, or parallel planar forms—S3D modules achieve substantial reductions in parameter count and computational cost while often maintaining or improving predictive performance on video analysis, volumetric image processing, and 3D vision tasks.

1. Mathematical Foundations of Separable 3D Convolutions

The canonical 3D convolution applies a learned kernel $$W\in\mathbb{R}^{k_t\times k_h\times k_w \times C_{\text{in}} \times C_{\text{out}}}$$ to an input tensor $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$, jointly mixing spatial, temporal, and channel dimensions:

$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$

Separable 3D convolutions approximate or decompose this operation via lower-rank or axis-wise factorizations:

• (a) Spatial–temporal factorization: Replace a $k_t\times k\times k$ 3D kernel by a $1\times k\times k$ spatial convolution $W_s$ followed by a $k_t\times 1\times 1$ temporal convolution $W_t$:

$Z = W_s * X;\quad Y = W_t * Z$

(Xie et al., 2017)

• (b) Depthwise separable variant: Factor into a per-channel $k^3$ 3D convolution (depthwise) plus a $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$0 channel-mixing convolution (pointwise):

$$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$1

(Ye et al., 2018, Rahim et al., 2021)

• (c) Orthogonal-plane separation (ACSConv): Concatenate 2D convolutional projections along three orthogonal planes:

$$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$2

Each branch uses a kernel of shape $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$3, $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$4, or $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$5 (Yang et al., 2019).

• (d) 1D-convolutional decomposition (3D-DSC): Decompose the kernel as a product of 1D convolutions along each spatial axis, with dense inter-layer connectivity to preserve expressiveness (Qu et al., 2019).
• (e) Parallel and multi-view separable blocks (PmSCn): Construct $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$6 parallel streams, each performing $$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$7 consecutive planar (2D) convolutions along orthogonal axes, followed by a 1D convolution per stream, to better cover multi-axis context (Gonda et al., 2018).

2. Parameter and Computational Complexity

Separable 3D convolutional variants exhibit dramatic improvements in model and compute efficiency versus standard 3D convolution:

Method	Params per Layer	FLOPs per Layer	Reduction vs Full 3D
Standard 3D Conv	$$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$8	$$X\in\mathbb{R}^{T\times H\times W\times C_{\text{in}}}$$9	—
Depthwise S3D	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$0	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$1	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$2 (typical)
ACSConv	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$3	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$4	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$5
3D-DSC (rank-$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$6)	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$7	$$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$8 (w/ $$Y_{t,h,w,c'} = \sum_{\tau=1}^{k_t} \sum_{i=1}^{k_h} \sum_{j=1}^{k_w} \sum_{c=1}^{C_\text{in}} X_{t+\tau-\Delta_t,\,h+i-\Delta_h,\,w+j-\Delta_w,\,c} \cdot W_{\tau,i,j,c,c'} + b_{c'}$$9)	$k_t\times k\times k$0 ($k_t\times k\times k$1)
PmSCn (typical)	$k_t\times k\times k$2	$k_t\times k\times k$3 (with $k_t\times k\times k$4 chosen)	up to $k_t\times k\times k$5 reduction
FDwSC	$k_t\times k\times k$6 + $k_t\times k\times k$7 + $k_t\times k\times k$8	—	up to $k_t\times k\times k$9 reduction

Note: Savings depend on channel dimension, kernel size ($1\times k\times k$0), and whether spatial/temporal splits are balanced.

3. Architectural Integration and Design Patterns

Separable 3D convolutions are flexibly integrated into diverse backbones:

• Inception-I3D S3D: Replace $1\times k\times k$1 convolutions in “Inception” modules with a $1\times k\times k$2 spatial kernel followed by a $1\times k\times k$3 temporal kernel. Optimal results (“top-heavy” S3D) are obtained when only the final two Inception modules use S3D, with preceding blocks using 2D-only operations (Xie et al., 2017).
• Depthwise S3D: Replace each Conv3D($1\times k\times k$4) layer in VGG/ResNet/U-Net with Depthwise3D($1\times k\times k$5) followed by Pointwise3D($1\times k\times k$6), plus normalization and nonlinearity (Ye et al., 2018, Rahim et al., 2021).
• ACSConv: Any 2D CNN (e.g., ResNet, DenseNet, DeepLab) can be converted by mapping 2D kernel weights to the three 2D-view branches, using unsqueezing and concatenation. 2D → 3D mappings of convolution and normalization layers are direct, allowing seamless weight transfer (Yang et al., 2019).
• 3D-DSC: Replace each standard 3D conv layer with one or more rank-$1\times k\times k$7 3D-DSC modules, comprising stacks of 1D convolutions (with dense connectivity and nonlinearities) and a $1\times k\times k$8 bottleneck (Qu et al., 2019).
• PmSCn: Replace single or stacked 3D conv layers with $1\times k\times k$9 parallel convolutional streams along different axes, each performing $W_s$0 consecutive 2D convolutions and a final 1D convolution, concatenating along the channel dimension (Gonda et al., 2018).
• Stereo cost-volumes (S3D in stereo): Replace each 3D conv in cost-aggregation with depthwise separable blocks (FwSC/FDwSC) for major compute reduction (Rahim et al., 2021).

4. Empirical Performance and Task Benchmarks

Separable 3D convolutions yield strong empirical performance across domains, with characteristic trends:

• Video Classification: On Kinetics-400, Inception-I3D achieves 71.1% Top-1 (107.9 GFLOPs), S3D 72.2% (66.4 GFLOPs), and S3D-G 74.7% (71.4 GFLOPs) (Xie et al., 2017). S3D-G attains the best accuracy-to-compute trade-off.
• 3D Vision (ShapeNetCore classification): S3D-VGG13 uses 1.17M conv params (95.8% fewer) with 95.10% accuracy (vs. 95.11% for standard) (Ye et al., 2018).
• Volumetric Reconstruction and Segmentation: S3D and P3D offer slightly reduced mIoU ($W_s$1 drop for S3D) but dramatic parameter savings vs. standard 3D decoders (Ye et al., 2018). 3D-DSC modules yield Dice coefficients up to 0.7932 on BRATS2017, outperforming standard U-Net and V-Net (Qu et al., 2019).
• Medical Imaging (ACSConv): On LIDC-IDRI, ACS-pretrained models achieve 76.5% Dice, 94.9% AUC—improving over both 2.5D and inflated 3D (I3D) models (Yang et al., 2019). On LiTS, ACS gives lesion global Dice 79.1% (vs. 76.5% for full 3D).
• Stereo Matching: Replacing 3D kernels with FwSC/FDwSC in GANet yields up to $W_s$2 reduction in operations, $W_s$3 in parameters, with equal or improved accuracy (e.g., 3-px error reduces from 4.21% to 3.94% on SceneFlow) (Rahim et al., 2021).

5. Comparative Analysis with Related Factorizations

Separable 3D convolutions are closely related to, but distinguished from, the following:

• Pseudo-3D (P3D) and (2+1)D: Decompose 3D kernel as a spatial $W_s$4 followed by $W_s$5; parameter savings are moderate ($W_s$6), and these do not directly enable weight transfer from 2D pretrained models (Xie et al., 2017, Ye et al., 2018, Yang et al., 2019).
• Depthwise vs. ACS vs. Parallel Streams: Depthwise approaches excel when high channel count or kernel size is present; ACSConv is optimal for leveraging 2D-frozen weights and compressing model size; parallel streams (PmSCn) empirically boost performance by aggregating multiple plane-wise contexts (Ye et al., 2018, Yang et al., 2019, Gonda et al., 2018).
• 1D-convolutional decomposition (3D-DSC): Achieves $W_s$7 parameter/FLOP reduction per block (for rank-1), supports deeper stacks, leverages nonlinearity and feature reuse, and is empirically validated on ADHD classification and BRATS brain tumor segmentation (Qu et al., 2019).

6. Best Practices, Limitations, and Recommendations

• Design strategy: In deep video models, “top-heavy” S3D designs—S3D modules only in high semantic layers, 2D blocks elsewhere—strike the best speed-to-accuracy balance (Xie et al., 2017). In resource-constrained or memory-limited settings, depthwise S3D variants are preferred (Ye et al., 2018). When maximal information transfer from large 2D corpora is required, use orthogonal-plane (ACS) variants (Yang et al., 2019).
• Limitations: Full 3D convolution may still be required for very low-level spatiotemporal analysis (e.g., arrow-of-time prediction) or when cross-channel interactions are core to the task—S3D variants may underfit such structure (Xie et al., 2017, Ye et al., 2018).
• Hardware considerations: Theoretical speed-ups may be offset by hardware memory access overheads for group-conv and depthwise implementations on some platforms (Ye et al., 2018).
• Plug-and-play integration: For most applications, S3D modules are drop-in replacements for Conv3D layers; no modifications to surrounding network or training regimes are necessary (Rahim et al., 2021, Ye et al., 2018, Yang et al., 2019).
• Feature gating: In video action recognition, channel-wise gating after temporal convolutions (“S3D-G”) further improves accuracy at moderate extra compute (Xie et al., 2017).

7. Impact and Empirical Summary

Separable 3D convolutions provide a principled, architecture-agnostic, and empirically validated approach to reducing the computational and memory footprint of 3D CNNs. They:

• Achieve up to $W_s$8 parameter and FLOP reductions (e.g., depthwise S3D (Ye et al., 2018)).
• Enable deeper 3D architectures and scaling to high-resolution volumetric data (e.g., classification, segmentation, reconstruction tasks (Ye et al., 2018, Qu et al., 2019, Gonda et al., 2018)).
• Routinely preserve, or even enhance, predictive accuracy across video, medical, and stereo-matching domains (Xie et al., 2017, Gonda et al., 2018, Yang et al., 2019, Rahim et al., 2021).
• Support efficient adaptation of 2D pretraining pipelines and multitask transfer (Yang et al., 2019).
• Serve as a preferred design for mobile/embedded scenarios and as a regularization mechanism against overfitting in limited-data settings.

Further developments are focused on data-driven determination of separation axes, adaptive multi-stream aggregation, and the integration of learned plane-weighting or attention for enhanced 3D context modeling. Continued comparative benchmarking in application-specific contexts is essential to refine best practices.