Conv-GRU: Convolutional GRU

• Conv-GRU is a spatiotemporal architecture that replaces fully-connected operations with convolutions, preserving spatial structure in sequential data.
• It models local dependencies via small convolutional kernels, significantly reducing parameter count compared to traditional GRUs.
• Integrated into deep networks, Conv-GRUs enhance tasks like video segmentation, denoising, and classification with improved metrics and efficiency.

A convolutional gated recurrent unit (Conv-GRU) is a spatiotemporal recurrent neural architecture that replaces the fully-connected operations in a standard gated recurrent unit (GRU) with convolutional transforms, thereby preserving and exploiting spatial structure in sequential data such as video, images, or spatial feature maps. This architecture enables efficient parameter sharing and spatially localized gating, and has been shown to improve performance in a range of video understanding, denoising, and sequential classification tasks by integrating both local spatial and temporal dependencies (Siam et al., 2016, Guo et al., 2022, Ballas et al., 2015, Jung et al., 2017, Golmohammadi et al., 2018).

1. Mathematical Definition and Formulation

Let $x_t \in \mathbb{R}^{H \times W \times C_{in}}$ be an input feature map at time $t$ and $h_{t-1} \in \mathbb{R}^{H \times W \times C_h}$ the previous hidden state. The Conv-GRU equations are as follows: $$z_t = \sigma(W_{xz} * x_t + W_{hz} * h_{t-1} + b_z)$$

$$r_t = \sigma(W_{xr} * x_t + W_{hr} * h_{t-1} + b_r)$$

$$\tilde{h}_t = \tanh(W_x * x_t + W_h * (r_t \odot h_{t-1}) + b_h)$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

where $*$ denotes 2D convolution, $\odot$ is elementwise product, and all bias terms $b_*$ are per-channel. For video denoising, activation functions such as ReLU can replace tanh in the candidate path (Guo et al., 2022).

All convolutional kernels $W_{*}$ have shape $$k \times k \times \text{in-channels} \times \text{out-channels}$$ (typically $k=3$), with stride 1 and padding to preserve feature map spatial resolution (Siam et al., 2016, Guo et al., 2022, Ballas et al., 2015, Jung et al., 2017, Golmohammadi et al., 2018).

2. Spatial Configuration and Parameter Efficiency

Conv-GRUs exploit spatial locality by sharing small convolutional kernels across the spatial domain rather than flattening the spatial grid. This means that each Conv-GRU gate applies a local convolution (e.g., $3 \times 3$) rather than a dense transform, so the total number of parameters per gate is linear in feature map channel dimensionality and kernel size, not in the overall spatial size:

• Input-to-hidden: $(k \times k \times C_{in} \times C_h)$
• Hidden-to-hidden: $(k \times k \times C_h \times C_h)$ per gate This results in orders of magnitude fewer parameters than a fully connected GRU operating on flattened maps (Ballas et al., 2015).

Critically, the spatial topology of the input is preserved through all Conv-GRU operations, enabling temporally recursive processing at each spatial location and facilitating efficient end-to-end training and inference in convolutional architectures (Siam et al., 2016).

3. Integration into Deep Architectures

Conv-GRU modules are typically integrated between the encoder and decoder of a convolutional backbone (such as VGG or ResNet):

• In video segmentation, the Conv-GRU is inserted after the last encoder convolution (prior to pixel-level classification), unrolled over a sliding window of $T$ frames, and followed by a segmentation head and upsampling (Siam et al., 2016).
• In hierarchical settings, Conv-GRUs can be applied over intermediate "percept" feature maps at multiple depths, processing each in parallel or with inter-layer feedback (Ballas et al., 2015).
• For single-stream sequential tasks, Conv-GRUs are applied after a convolutional front-end and before any fully connected or softmax layers (e.g., for EEG or speech) (Golmohammadi et al., 2018).

Conv-GRUs can be used in both unidirectional and bidirectional recurrent setups, and may be stacked or combined with convolutional LSTMs or normalization modules (Ballas et al., 2015, Jung et al., 2017).

4. Training Methodologies and Optimization

Conv-GRU-based models are trained end-to-end using backpropagation through time (BPTT), typically with objective functions appropriate to the downstream task:

• Pixel-wise cross-entropy loss for segmentation tasks, optimized with Adadelta or similar optimizers (Siam et al., 2016).
• Weighted $L_1$ or $L_2$ losses for denoising tasks, targeting both immediate and fused outputs (Guo et al., 2022).
• Mean squared error for sequence classification (Golmohammadi et al., 2018).

Initialization is crucial: Orthogonal or well-scaled initializations are necessary for stable training, while bias initialization (notably setting update-gate biases to negative values like $-2$) is recommended to prevent vanishing outputs during early training (Jung et al., 2017, Golmohammadi et al., 2018).

Regularization strategies applied include dropout on convolutional outputs, weight decay (L1, L2, or both), and occasionally additive Gaussian noise, especially in the convolutional front-end layers (Golmohammadi et al., 2018).

Adaptive Detrending (AD) is an additional temporal normalization scheme, interpreting the hidden state $h_t$ as an adaptive trend and subtracting it from the candidate $\tilde{h}_t$ before forwarding to the next layer, which accelerates training and improves generalization with virtually no additional computational cost (Jung et al., 2017).

5. Comparative Performance Across Domains

Empirical evaluation demonstrates that Conv-GRU architectures deliver consistent improvements over analogous non-recurrent or fully connected-recurrent baselines:

• In video segmentation, Conv-GRU integration improves F-measure on SegTrack V2 by $+5.2\%$, DAVIS by $+3.8\%$, mean IoU on SYNTHIA by $+5.7\%$, and categorical IoU on CityScapes by $+3.5\%$. Largest gains are realized for moving object classes (Siam et al., 2016).
• In video denoising, Conv-GRU-based GRU-VD outperforms methods such as EDVR and RViDeNet, achieving PSNR of 45.06 dB and SSIM of 0.9981 on the CRVD benchmark (Guo et al., 2022).
• For action recognition and video captioning, Conv-GRUs yield gains of $1.9$–$3.4\%$ absolute accuracy and $+10\%$ BLEU-4 in captioning over baselines, matching or slightly exceeding more complex models like ConvLSTM at lower computational cost (Ballas et al., 2015).
• In EEG seizure detection, Conv-GRU achieves 91.49% specificity at 30.8% sensitivity, albeit with more false positives than Conv-LSTM under matched settings (Golmohammadi et al., 2018).
• In long-range contextual video recognition, Conv-GRU with AD normalization achieves up to $98.5\%$ accuracy, converges twice as fast as non-detrended Conv-GRUs, and generalizes better than 3D CNNs or feed-forward spatial networks (Jung et al., 2017).

Most ablations conclude that the empirical gains are attributable to spatiotemporal modeling rather than mere parameter count increases, as extra convolutional layers alone do not match the improvements seen with recurrent gating (Siam et al., 2016).

6. Domain-Specific Adaptations and Extensions

Conv-GRU has been adapted for several domain-specific challenges:

• Video Denoising: Augments standard gating with the injection of estimated per-frame noise standard deviation through 1x1 convolutions into each gate, and employs IMDN-based sub modules for robust artifact suppression under varying illumination (Guo et al., 2022).
• Temporal Normalization: Adaptive Detrending operates in Conv-GRUs by subtracting the trend (EMA over time with instantaneous decay) per neuron, stabilizing activations over long video sequences (Jung et al., 2017).
• Stacked and Multi-level Designs: Hierarchical Conv-GRUs process multi-scale CNN features ("percepts"), with optional inter-layer recurrence, facilitating rich spatiotemporal fusion for complex sequence decoding (Ballas et al., 2015).
• Comparison to ConvLSTM: Conv-GRU architectures are 20–30% more parameter-efficient than ConvLSTM owing to one fewer gating path, with modest differences in accuracy depending on task requirements for explicit memory (Ballas et al., 2015, Golmohammadi et al., 2018).

7. Practical Insights and Implementation Considerations

For robust integration and effective training of Conv-GRUs:

• Preserve spatial resolution via "same" convolutions (padding of one for 3x3 kernels).
• Select hidden state depth ($C_h$) to balance expressivity and memory; typical values are 64–256.
• Use bias initialization (update gate to negative values) and orthogonal kernel initialization.
• Utilize standard normalization (batch-norm, layer-norm) in convolutional front ends, while AD provides complementary temporal normalization.
• Employ end-to-end training with BPTT, optimize using SGD with momentum for vision tasks, and apply early stopping based on validation loss plateau.

Conv-GRU is broadly applicable to any task requiring spatiotemporal modeling in grid or image-like data, including video segmentation, recognition, denoising, captioning, and sequential biomedical analysis (Siam et al., 2016, Guo et al., 2022, Ballas et al., 2015, Jung et al., 2017, Golmohammadi et al., 2018).