Convolutional GRU for Spatiotemporal Modeling

• Conv-GRU is a recurrent unit that replaces dense matrix multiplications with convolutions, preserving spatial feature maps and local connectivity.
• It is widely applied in dense-prediction systems such as video segmentation, BEV perception, and contextual video recognition, yielding better temporal consistency.
• Extensions like geometry-aware masks, adaptive detrending, and diffusion convolutions further improve training stability and computational efficiency.

Searching arXiv for recent and foundational Conv-GRU papers to ground the article. I’ll check for arXiv entries on Conv-GRU, recurrent FCNs, and graph/diffusion GRUs. A Convolutional Gated Recurrent Unit (Conv-GRU, or ConvGRU) is a GRU variant in which the dense affine transforms of the recurrent update are replaced by convolutions, so that both the input $x_t$ and the hidden state $h_t$ remain spatial tensors rather than flattened vectors. In the canonical formulation, the reset gate, update gate, candidate state, and hidden-state update are computed directly on feature maps, which preserves spatial connectivity, reduces parameter growth relative to fully connected recurrence on images, and makes the unit suitable for spatiotemporal modeling in video, Bird’s-Eye View (BEV) perception, and related dense-prediction settings (Siam et al., 2016). The term is, however, used inconsistently across the literature: several papers labeled “convolutional GRU” are actually CNN+GRU hybrids in which convolution is confined to a front-end encoder and the recurrent stage is a standard GRU over a 1D sequence of vectors or phrase features rather than a recurrent convolution on feature maps (Lyu et al., 2018).

1. Formal definition and computational rationale

The defining modification from a standard GRU is the substitution of matrix multiplications by convolutions. In the recurrent fully convolutional network for video segmentation, the Conv-GRU is written as

$$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$

$$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$

$h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$

Here, $x_t$ is a spatial feature tensor extracted from frame $t$, $h_t$ is a hidden-state feature map, $\ast$ denotes convolution, and $h_t$0 is elementwise multiplication. The same structural idea appears in multi-level video representation learning, where GRU gate and candidate computations are performed with zero-padded $h_t$1D convolutions on CNN percept maps rather than dense maps on flattened features (Siam et al., 2016, Ballas et al., 2015).

The computational motivation is twofold. First, flattening a feature map destroys explicit spatial arrangement. Second, a conventional recurrent unit on an image-like tensor leads to parameter growth tied to spatial extent. The video segmentation paper states that for a feature map of spatial size $h_t$2 and channels $h_t$3, a conventional recurrent unit requires on the order of

$h_t$4

weights, whereas Conv-GRU uses shared local kernels of size

$h_t$5

The multi-level video representation paper makes the same argument in a different notation: input-to-hidden parameters scale as $h_t$6 rather than $h_t$7, with corresponding savings in computation and storage (Siam et al., 2016, Ballas et al., 2015).

This replacement is not merely an implementation detail. It changes the semantics of the recurrent state. In a standard GRU, $h_t$8 is typically a vector. In Conv-GRU, $h_t$9 is a spatial feature map whose location $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$0 depends on a local neighborhood in the current input map and previous hidden map. With zero-padding and stride $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$1, spatial dimensions are preserved through recurrent updates, so temporal memory remains aligned to image or BEV coordinates (Ballas et al., 2015, Jung et al., 2017).

2. Architectural placement in dense-prediction systems

Conv-GRU is typically not used in isolation. It is inserted into a larger convolutional pipeline at the level of intermediate feature maps. In recurrent fully convolutional video segmentation, each frame is first processed by a fully convolutional frontend, after which Conv-GRU receives either a coarse heat map or, in the best-performing convolutional recurrent models, an intermediate feature map. In the RFC-VGG configuration, input frames of size $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$2 are passed through a reduced VGG-F frontend, then a ConvGRU $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$3, 128 channels operates on the resulting 128-channel intermediate feature map; a $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$4 convolution produces a coarse segmentation map, and a deconvolution upsamples to dense output for the last frame in the window (Siam et al., 2016).

The same principle appears in multi-level recurrent video representation learning. There, Conv-GRU is applied not only to top features but to percepts from pool2, pool3, pool4, pool5, and fc7 of VGG-16. The recurrent streams operate on feature maps at multiple depths, and the last hidden representations are pooled and classified. The paper also studies a stacked variant in which recurrent layers at higher CNN levels receive current-time hidden states from lower levels, using max-pooling between recurrent layers to align spatial sizes (Ballas et al., 2015).

In BEV segmentation, the recurrent insertion point moves from image space to projected BEV space. Geo-ConvGRU is embedded in a four-stage pipeline consisting of a backbone, a BEV projection module, a Geo-ConvGRU temporal module, and a prediction head. Each timestep contains six monocular camera views with inputs

$$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$5

where $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$6, $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$7, $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$8, and $$z_t = \sigma(W_{hz}\ast h_{t-1} + W_{xz}\ast x_t + b_z),$$9. A $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$0D image backbone, instantiated as EfficientNet-B4 in experiments, extracts image features; Lift-Splat-Shoot/FIERY-style depth-based lifting projects them into BEV; and ConvGRU fuses the resulting BEV features temporally before the final head predicts semantic segmentation, perceived maps, or future instance segmentation. The BEV output resolution in the main setup is $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$1, corresponding to a $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$2 map at $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$3 cm resolution (Yang et al., 2024).

The common architectural role across these systems is therefore stable: a per-frame or per-step convolutional encoder extracts spatial features, Conv-GRU accumulates temporal evidence while preserving topology, and a task head decodes the final hidden representation.

3. Extensions and specialized recurrent formulations

Several papers extend the basic Conv-GRU mechanism rather than altering its core gate logic. Geo-ConvGRU retains standard ConvGRU over BEV features,

$$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$4

$$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$5

$$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$6

but then applies a geometry-derived geographical mask,

$$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$7

where valid voxels receive weight $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$8 and invalid voxels receive $$r_t = \sigma(W_{hr} \ast h_{t-1} + W_{xr} \ast x_t + b_r),$$9, set to $$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$0 in all experiments. The mask is derived from camera geometry via the projection

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$1

and is intended to suppress temporal noise caused by invalid BEV voxels, such as “ghost” vehicles persisting from prior frames (Yang et al., 2024).

A different extension addresses optimization rather than recurrence semantics. Adaptive detrending (AD) interprets the GRU hidden-state update as an exponential moving average and treats the hidden state itself as an adaptive trend estimate. For ConvGRU, the detrended output is

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$2

which is passed onward instead of the raw hidden activation. The paper frames AD as temporal normalization: it removes “internal covariate shift within a sequence of each neuron” by subtracting a learned trend, adds almost no extra computational or memory cost, and is designed specifically to accelerate ConvGRU training for contextual video recognition (Jung et al., 2017).

A graph-structured generalization replaces Euclidean convolutions with diffusion convolutions on sensor networks. In Diffusion Convolutional GRU (DCGRU), the gate transforms are written as

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$3

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$4

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$5

$$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$6

Here the “convolution” is a diffusion process over a graph rather than a regular-grid kernel, so DCGRU is best regarded as a graph-convolutional analogue of Conv-GRU rather than a standard image-space Conv-GRU (Dong et al., 2024).

A restoration-oriented reformulation appears in video denoising. GRU-VD preserves GRU logic but operates directly on the previous denoised output $$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$7 rather than on an abstract hidden tensor, uses $$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$8 and the current-frame noise standard deviation $$\hat{h_t} = \Phi(W_h \ast (r_t \odot h_{t-1}) + W_x \ast x_t + b),$$9 in the reset gate, replaces $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$0 by ReLU in the candidate computation, and supervises both the intermediate estimate $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$1 and the final output $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$2 through

$h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$3

with $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$4 and $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$5. This is not presented as a generic Conv-GRU, but it is a clear task-specific convolutional reinterpretation of GRU recurrence for image-space video restoration (Guo et al., 2022).

4. Empirical performance and application domains

In video segmentation, recurrent fully convolutional networks consistently improve single-frame baselines. On SegTrack V2, FC-VGG achieves F-measure 0.7254 whereas RFC-VGG reaches 0.7767; on DAVIS, FC-VGG yields 0.6066 and RFC-VGG 0.6304. For semantic segmentation on Synthia, FC-VGG mean class IoU is 0.755 and RFC-VGG is 0.812; on CityScapes, FCN-8s category IoU is 0.53 and RFCN-8s is 0.565. An important control is FC-VGG Extra Conv, whose SegTrack V2 F-measure of 0.7493 remains below RFC-VGG 0.7767, supporting the interpretation that temporal recurrence, not merely extra capacity, drives the gain (Siam et al., 2016).

In contextual video recognition, the advantage of ConvGRU becomes stronger as temporal reasoning requirements grow. On the OA dataset, a spatial CNN reaches 59.8% joint accuracy, C3D 97.2%, and the ConvGRU baseline about 97.0%. On the more context-dependent OA-M dataset, the spatial CNN falls to 26.4%, C3D reaches 68.8%, and the ConvGRU baseline attains 92.9%. With adaptive detrending, ConvGRU rises to 98.5% on OA and 95.4% on OA-M; LN+AD further reaches 97.2% on OA-M (Jung et al., 2017).

In BEV perception, replacing a 3D-CNN temporal block with ConvGRU improves long-range temporal modeling. Against FIERY’s temporal module, Geo-ConvGRU improves semantic segmentation under all three evaluation settings on nuScenes: 38.2 IoU $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$6 39.5, 39.9 $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$7 41.7, and 57.6 $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$8 59.3. For perceived maps prediction, average IoU improves from 40.2 for FIERY to 42.1 for Geo-ConvGRU; for future instance segmentation over 2.0 seconds, the method reaches 37.7 future semantic IoU, 29.8 PQ, 70.3 SQ, and 42.7 RQ. The ablation at $h_t = (1-z_t)\odot h_{t-1} + z \odot \hat{h_t}.$9 is especially diagnostic: 3D CNN gives long IoU 37.7, ConvGRU 38.2, and Geo-ConvGRU 38.8, suggesting that standard ConvGRU already improves long-range temporal dependency modeling and the geographical mask yields a further gain on long-range metrics (Yang et al., 2024).

In graph-structured pedestrian forecasting, DCGRU outperforms plain GRU, and the DTW-augmented graph improves DCGRU further. With $x_t$0, DCGRU-DTW achieves about 1.3% lower MAPE than DCGRU at 1h ahead, and 3–4% lower MAPE at 3h–5h ahead. With $x_t$1, DCGRU-DTW remains best; at 5h ahead, compared with DCGRU, MAE is about 5 units smaller, MAPE 3.5% smaller, and RMSE about 10 units smaller. This suggests that Conv-GRU-like recurrence remains effective when the convolution operator is transferred from Euclidean grids to diffusion on irregular graphs (Dong et al., 2024).

5. Terminological ambiguity and common misconceptions

A recurring source of confusion is that “convolutional GRU” is often used for architectures that are not true Conv-GRU cells. The distinction is architectural rather than cosmetic.

Formulation	Recurrent transform	Representative papers
True Conv-GRU	GRU gates use convolutions on spatial feature maps	(Siam et al., 2016, Jung et al., 2017, Yang et al., 2024)
CNN+standard GRU hybrid	CNN outputs vectors or phrase features; GRU is standard	(Lyu et al., 2018, Golmohammadi et al., 2018, He et al., 2018)
Diffusion-convolutional GRU	GRU gates use graph diffusion convolution	(Dong et al., 2024)

The road-segmentation paper is explicit evidence for the hybrid case. Its input is a $x_t$2 tensor, the CNN encoder converts this into 60 feature vectors, each 256-D, and the recurrent stage is a bidirectional GRU with 128 hidden units per direction scanning horizontally over a 1D spatial sequence. The paper itself should therefore be classified as CNN+GRU spatial-sequence segmentation, not as a true Conv-GRU, because the recurrent state is a vector sequence and the gate transforms are dense matrices rather than convolutions (Lyu et al., 2018).

The same caveat applies in other modalities. In seizure detection, the authors’ “convolutional GRU network” consists of LFCC-based inputs, three layers of 2D convolution and pooling, a 1D temporal convolution and pooling, and then a deep bidirectional GRU over a reduced temporal sequence of 26 time steps. In medical relation classification, “Convolutional Gated Recurrent Units” denotes a CNN $x_t$3 BiGRU text model in which a width-3 convolution produces phrase vectors $x_t$4 and a standard bidirectional GRU models their dependencies. In LIBS concentration prediction and streamlined-weir discharge prediction, “Conv-GRU” similarly refers to a convolutional front-end followed by GRU layers, with no recurrent convolution inside the gate equations (Golmohammadi et al., 2018, He et al., 2018, Rezaei et al., 2023, Chen et al., 2022).

This terminological instability has a practical consequence. A citation to a “convolutional GRU” paper does not, by itself, establish that recurrent gates are convolutional. For image- or BEV-space Conv-GRU, the decisive criterion is whether $x_t$5 and $x_t$6 are replaced by convolutional operators acting on spatial tensors; for graph-structured variants, the analogue is whether they are replaced by graph diffusion convolutions (Yang et al., 2024, Dong et al., 2024).

6. Optimization, limitations, and research directions

Conv-GRU improves spatially structured sequence modeling, but training and deployment trade-offs remain central. The contextual video recognition study characterizes ConvGRU training as difficult and slow enough to motivate a dedicated temporal-normalization method. Adaptive detrending is proposed because ConvGRU hidden dynamics exhibit temporal internal covariate shift, and the paper argues that detrending via the hidden state accelerates convergence and improves generalization. A practically important detail is update-gate bias initialization: initializing the update gate bias to $x_t$7 rather than $x_t$8 speeds convergence for both baseline ConvGRU and AD-enhanced ConvGRU, because an initially smaller update gate prevents the hidden trend from tracking the candidate too closely and preserves a stronger residual signal (Jung et al., 2017).

The BEV literature emphasizes a different trade-off: temporal field length versus runtime. Geo-ConvGRU evaluates $x_t$9, $t$0, and $t$1. The strongest practical setting is Geo-ConvGRU with $t$2, with train time 26.2 h, 4.9 FPS, short IoU 68.6, long IoU 39.5, short PQ 59.9, and long PQ 31.9. At $t$3, accuracy rises slightly further, but speed drops to 3.3 FPS and training time to 33.9 h. The paper presents this as evidence that ConvGRU scales better with temporal field than 3D CNN, while still remaining materially more efficient than BEVFormer at 64.3 h and 1.9 FPS for $t$4 (Yang et al., 2024).

Reporting incompleteness is another recurrent limitation. The video segmentation paper does not clearly specify the temporal window length or hidden-state initialization in the provided text. Geo-ConvGRU does not state the exact ConvGRU kernel size or hidden-state channel width. Several hybrid papers using the term “convolutional GRU” omit kernel sizes, stride, padding, dropout, or batch size, which complicates exact reproduction even when the high-level architecture is clear (Siam et al., 2016, Yang et al., 2024, Rezaei et al., 2023). A plausible implication is that “Conv-GRU” functions as a family resemblance term across subfields rather than a uniformly specified cell class.

Across the cited literature, three research directions recur. One is geometry-aware or visibility-aware recurrent masking, exemplified by Geo-ConvGRU. A second is training stabilization through temporal normalization, exemplified by adaptive detrending. A third is domain-specific reparameterization of the recurrence, as in image-space denoising and diffusion-convolutional forecasting. Despite these variations, the core identity of Conv-GRU remains stable: a GRU whose recurrence acts on structured spatial representations by replacing dense maps with convolutional ones, thereby coupling temporal memory with local spatial topology (Jung et al., 2017, Yang et al., 2024, Dong et al., 2024).