ConvRNN: Convolutional Recurrent Neural Networks

• ConvRNNs are neural architectures that combine convolutional operators with recurrent dynamics to integrate spatial detail and temporal context.
• They encompass diverse designs like embedded recurrence and sequential coupling, using gated units and parameter-efficient methods to stabilize learning.
• Applications span robust vision, speech enhancement, language translation, and tactile processing, offering practical trade-offs in accuracy and computational cost.

Convolutional recurrent neural networks (ConvRNNs) are neural architectures that combine convolutional operators with recurrent state evolution. In the literature, the term denotes several non-identical constructions: convolutional layers whose activations are recurrently updated across time, recurrent parameterizations of convolution itself, channel-wise recurrent simulations of wide convolutions, and multi-stage pipelines in which convolutional front ends feed recurrent sequence models. Despite this heterogeneity, the unifying objective is consistent: to couple the locality and parameter sharing of convolution with the context accumulation, temporal sensitivity, or compositional modeling of recurrent computation across domains including vision, speech, language, wireless signal processing, translation, and tactile sensing (Hartmann, 2018, Retsinas et al., 2019, Zhao et al., 2018, Mallick et al., 2021, Chung et al., 23 May 2025).

1. Terminological scope and architectural families

A useful distinction is between “embedded recurrence” (Editor’s term) and “sequential coupling” (Editor’s term). In embedded-recurrence models, the convolutional operator itself is recurrent. Examples include the gruCNN, which replaces a classical $3\times3$ convolution by a gated recurrent convolution over a hidden-state tensor $h_t$ (Hartmann, 2018); the Recurrent Convolutional Network (RCN), which decomposes a 3D convolution into a 2D spatial convolution plus a recurrent $1\times1\times1$ temporal convolution (Singh et al., 2018); Channel-wise Recurrent Convolution (CRC), which processes channel segments recurrently and concatenates the resulting hidden states (Retsinas et al., 2019); and the Gated Recurrent Convolutional Layer (GRCL), which modulates recurrent context by multiplicative gates (Wang et al., 2021).

In sequentially coupled models, convolutional and recurrent modules are composed in stages rather than fused into a single recurrent convolutional operator. “Convolutional-Recurrent Neural Networks for Speech Enhancement” uses a 2D convolutional front end, a deep bidirectional LSTM, and a frame-wise fully connected output layer for spectrogram regression (Zhao et al., 2018). “Sequential Convolutional Recurrent Neural Networks for Fast Automatic Modulation Classification” applies two 1D convolutional layers before a two-layer LSTM stack (Liao et al., 2019). “Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation” places stacked 1D convolutional encoding layers before a bidirectional GRU encoder (Mallick et al., 2021). “ConvRNN-T” augments an LSTM-based RNN-T with local and global causal CNN encoders (Radfar et al., 2022).

The scope of the term also extends beyond temporal sensor streams. “Convolutional Neural Networks with Recurrent Neural Filters” models convolution filters with RNNs so that the filter itself captures language compositionality and long-term dependencies, arguing that the usual affine-plus-nonlinearity filter is inadequate for such structure (Yang, 2018). This broad usage indicates that ConvRNN is not a single canonical layer type but a family of mechanisms for introducing recurrent computation into convolution-centered architectures.

2. Representative recurrent-convolution operators

One influential formulation is the gated recurrent convolution used in the gruCNN. For each time step $t$, input feature map $x_t$, and hidden-state tensor $h_{t-1}$, the layer computes

$z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$

$r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$

$$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$

$$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$

with all convolutions same-padded and stride $h_t$0. Here the recurrence is spatially local and temporally persistent, and the hidden state is propagated at every spatial location (Hartmann, 2018).

RCN uses a simpler recurrent convolutional unit. At layer $h_t$1, with current spatial feature map $h_t$2 and previous hidden state $h_t$3,

$h_t$4

The temporal operator is a $h_t$5 hidden-state convolution, while the input pathway is a purely spatial $h_t$6D convolution. This factorization is explicitly designed to produce causal outputs, preserve temporal resolution, and avoid the anti-causal behavior of standard 3D convolutions (Singh et al., 2018).

CRC layers in RecNets make the recurrence run across channel segments rather than time. If $h_t$7 is split into contiguous channel blocks $h_t$8, then

$h_t$9

and the layer output is $1\times1\times1$0. Because $1\times1\times1$1, $1\times1\times1$2, and $1\times1\times1$3 are shared across recurrent steps, CRC reduces parameters relative to a standard convolution while simulating a wide transformation (Retsinas et al., 2019).

GRCL introduces explicit gating on the recurrent pathway. In its basic form,

$1\times1\times1$4

with

$1\times1\times1$5

The stated motivation is that a vanilla recurrent convolutional layer expands receptive fields unboundedly as recurrent iterations increase, whereas gating makes receptive fields adaptive to the input content and iteration (Wang et al., 2021).

A distinct theoretical construction appears in “Convolutional unitary or orthogonal recurrent neural networks.” There the recurrent operator is parameterized through a convolutional exponential

$1\times1\times1$6

equivalently

$1\times1\times1$7

If $1\times1\times1$8 is antisymmetric in the real case or anti-Hermitian in the complex case, then $1\times1\times1$9 is orthogonal or unitary, respectively. The paper’s central claim is that FFT-based forward and backward algorithms make this parametrization asymptotically no more expensive than the network’s iteration (Magnasco, 2023).

Collectively, these formulations show that ConvRNN research varies along at least three axes: what carries the state, where recurrence is inserted, and whether gating or constrained parameterizations are used to stabilize information flow.

3. Sequentially coupled convolution-plus-recurrence models

A large part of the literature uses convolution to produce locally structured features and recurrence to model longer-range dependence over the resulting sequence. In speech enhancement, the model summarized in “Convolutional-Recurrent Neural Networks for Speech Enhancement” takes a noisy spectrogram $t$0 with $t$1 frequency bins and $t$2 frames, applies a single 2D convolutional layer with $t$3 kernels of size $t$4, then feeds the resulting $t$5-dimensional per-frame features into a two-layer bidirectional LSTM with $t$6 hidden units per direction per layer, followed by a nonnegative frame-wise regressor. The model is trained end-to-end with mean squared error over the spectrogram target and reconstructs waveform by inverse STFT using the noisy phase (Zhao et al., 2018).

The text-classification CRNN described in “Combine Convolution with Recurrent Networks for Text Classification” assigns these roles differently. A CNN over the word-embedding matrix $t$7 produces feature matrix $t$8, which is converted into an aspect-wise weight matrix

$t$9

In parallel, a bidirectional GRU processes the sentence, and a neural tensor layer fuses forward and backward hidden states into $x_t$0. The final representation is

$x_t$1

which is flattened and classified with a fully connected layer and softmax. In this construction, convolution provides a learned 2D weighting over words, while recurrence supplies contextualized token representations (Lyu et al., 2020).

The neural machine translation encoder of “Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation” also follows a staged design. Source tokens receive word and position embeddings, then pass through $x_t$2 1D convolutional layers of width $x_t$3, each followed by a residual addition and layer normalization,

$x_t$4

after which the normalized sequence is fed into a bidirectional GRU with hidden size $x_t$5. The decoder is an attention-based recurrent network trained by negative log-likelihood over target tokens (Mallick et al., 2021).

In raw-signal settings, the same pattern appears with 1D convolutions. SCRNN for automatic modulation classification takes raw IQ inputs of shape $x_t$6, applies two 1D convolutional layers with $x_t$7 filters and kernel size $x_t$8, reshapes the result into a length-$x_t$9 sequence of $h_{t-1}$0-dimensional vectors, and processes it with two LSTM layers of $h_{t-1}$1 units before an $h_{t-1}$2-way softmax classifier (Liao et al., 2019). TCRN for waveform-level speech enhancement repeats blocks composed of 1D temporal convolution, batch normalization, PReLU, a uni-directional LSTM, and a transposed convolution, with residual shortcuts around both the LSTM and the entire block (Li et al., 2020).

ConvRNN-T occupies an intermediate position between tightly embedded and loosely coupled designs. Its local CNN encoder uses four layers of causal 2D convolution; its global encoder uses six causal 1D depthwise-separable blocks with dilation, squeeze-and-excitation, dropout, and residual connection; and these outputs are concatenated and projected before a seven-layer uni-LSTM acoustic encoder inside the RNN-T framework (Radfar et al., 2022). This suggests that the practical boundary between “convolutional recurrent network” and “convolution-augmented recurrent network” is often architectural rather than categorical.

4. Temporal integration, causality, and receptive-field control

One recurring motivation for ConvRNNs is the limitation of static convolutional receptive fields. RCN is explicit on this point: standard 3D CNNs are anti-causal, constrain temporal reasoning to the temporal kernel size, and are not temporal-resolution-preserving for sequence-to-sequence video modeling. By replacing each 3D convolution with a recurrent convolutional unit, RCN produces one output per frame, depends only on current and past frames, and can be unrolled over longer sequences at test time to extend the effective temporal horizon (Singh et al., 2018).

The gruCNN reaches a similar conclusion from a noise-robustness perspective. It treats video as a sequence of frames and embeds recurrence directly in convolutional layers at every spatial scale. On CIFAR-10 sequences formed by jittered copies of a static image over $h_{t-1}$3 training frames and $h_{t-1}$4 testing frames, the model’s recurrence is described as enabling local temporal integration early in the pipeline. The paper further fits framewise accuracy curves by

$h_{t-1}$5

using $h_{t-1}$6 as an integration-time constant, and reports that longer effective integration time at low SNR is associated with improved low-light performance (Hartmann, 2018).

GRCNN addresses the same issue through adaptive gating of recurrent context. The paper argues that in a plain recurrent convolutional layer, receptive fields expand unboundedly with recurrent iterations and may cause over-smoothing or loss of spatial specificity. GRCL therefore computes gate maps $h_{t-1}$7 that modulate the recurrent contribution. Empirically, gates in lower layers are reported to be more open, whereas deeper gates have smaller mean and higher variance, thereby focusing more selectively on the input content (Wang et al., 2021).

Other works make receptive-field growth a design variable. RecNets state that a CRC layer unrolled over $h_{t-1}$8 steps with $h_{t-1}$9 kernels accumulates a receptive field of approximate size $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$0, while preserving parameter sharing across recurrent steps (Retsinas et al., 2019). TCRN derives the waveform receptive field of a stack of temporal convolutional recurrent blocks as

$z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$1

and notes that with four blocks using $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$2 and $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$3, the receptive field covers hundreds of milliseconds of context (Li et al., 2020).

For streaming speech recognition, ConvRNN-T makes causality a system-level property. All convolutions are left-padded to preserve causality, the acoustic and prediction networks are uni-directional LSTMs, the global CNN encoder uses dilation up to $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$4 frames, and the reported end-to-end latency is approximately $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$5 ms plus LSTM thread time (Radfar et al., 2022). A plausible implication is that ConvRNN design frequently trades parallelism for precisely controlled causal context.

5. Training objectives, computational trade-offs, and reported behavior

Training objectives in ConvRNN work vary sharply with the target task. The gruCNN is trained with frame-wise cross-entropy averaged over $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$6 frames per sequence using RMSProp with initial learning rate $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$7, decay $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$8, and batch size $z_t = \sigma ( W_{zh} * h_{t-1} + W_{zx} * x_t ),$9 sequences (Hartmann, 2018). EHNet for speech enhancement uses end-to-end mean squared error on spectrograms and AdaDelta with scheduled learning rates $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$0 every $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$1 epochs (Zhao et al., 2018). The text-classification CRNN uses cross-entropy with Adam at learning rate $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$2, dropout on $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$3, $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$4, and $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$5, and $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$6 regularization (Lyu et al., 2020). The NMT encoder is optimized by negative log-likelihood with AdaDelta, mini-batch size $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$7, and early stopping on a validation split (Mallick et al., 2021). TCRN combines waveform MSE with two spectral-magnitude losses at STFT window lengths $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$8 and $r_t = \sigma ( W_{rh} * h_{t-1} + W_{rx} * x_t ),$9, using coefficient $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$0, batch normalization, PReLU, residual skips, Xavier initialization, weight decay $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$1, and gradient clipping to norm $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$2 in recurrent layers (Li et al., 2020).

The computational trade-off is equally central. In the gruCNN, each recurrent convolutional layer replaces one $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$3 kernel by multiple recurrent kernels, but channel width is reduced by approximately $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$4 to keep total parameters comparable: the cCNN has approximately $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$5 M parameters and the gruCNN approximately $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$6 M. The cost is layerwise FLOPs that grow by approximately $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$7 per recurrent layer and inference time that increases by a factor of $$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$8–$$\tilde h_t = \tanh ( W_{hh} * ( r_t \circ h_{t-1} ) + W_{hx} * x_t ),$$9 per frame (Hartmann, 2018). RecNets derive the CRC parameter count as $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$0, compared with $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$1 for a standard convolution, and state that the reduction is roughly $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$2, but also emphasize that the recurrent steps must be executed sequentially, limiting parallelism (Retsinas et al., 2019). ConvRNN-T reports encoder FLOPs of approximately $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$3 GFLOPs for $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$4 frames, compared with approximately $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$5 GFLOPs for a Conformer encoder, and total parameter count of approximately $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$6 M (Radfar et al., 2022). SCRNN reports $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$7 M parameters, approximately $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$8 reduction in total training time relative to an LSTM baseline, and approximately $$h_t = z_t \circ h_{t-1} + (1-z_t) \circ \tilde h_t,$$9 reduction in prediction time per sample (Liao et al., 2019).

Within-task empirical results are consistently reported as the main justification for these designs. At low SNR $h_t$00 and after $h_t$01 frames, the gruCNN achieves $h_t$02 accuracy, compared with $h_t$03 for a cCNN with Bayes-optimal frame-wise integration and $h_t$04 for a cCNN without Bayes integration; false rejection drops to $h_t$05, compared with $h_t$06 and $h_t$07, respectively (Hartmann, 2018). EHNet reports PESQ improvement of $h_t$08 on seen noise and $h_t$09 on unseen noise, with WER changing from $h_t$10 to $h_t$11 on seen noise and from $h_t$12 to $h_t$13 on unseen noise (Zhao et al., 2018). SCRNN reaches $h_t$14 classification accuracy at high SNR and outperforms the CNN baseline by $h_t$15 and the LSTM baseline by $h_t$16 at high SNR (Liao et al., 2019). ConvRNN-T reports LibriSpeech greedy-decoding WERs of $h_t$17 on test-clean and $h_t$18 on test-other, outperforming vanilla RNN-T, Conformer, and ContextNet under the reported setup (Radfar et al., 2022). RCN with a ResNet-18 backbone reports $h_t$19 clip-level and $h_t$20 video-level Top-1 accuracy on Kinetics, versus $h_t$21 and $h_t$22 for the corresponding I3D baseline, while on MultiThumos it reports $h_t$23 mAP@1 and $h_t$24 for the unrolled variant (Singh et al., 2018).

6. Domain impact, neuroscientific alignment, and unresolved questions

ConvRNNs have been adopted in domains where local structure and temporally extended evidence must be combined under noise, latency, or hierarchical constraints. In robust vision, recurrent convolutions are proposed for low-light or noisy video and for object recognition, detection, and scene text recognition (Hartmann, 2018, Wang et al., 2021). In speech and audio, they appear as spectrogram-to-spectrogram enhancers, end-to-end waveform enhancers, streaming ASR encoders, and emotion-recognition models with convolutional front ends and BLSTM back ends (Zhao et al., 2018, Li et al., 2020, Radfar et al., 2022, Huang et al., 2017). In language and translation, they are used to introduce compositional or phrase-level structure before recurrent sequence modeling (Yang, 2018, Mallick et al., 2021). In wireless signal analysis, they provide a compact convolutional distillation stage before temporal classification (Liao et al., 2019).

A particularly recent extension appears in tactile neuroscience and embodied AI. Chung et al. introduce an Encoder–Attender–Decoder framework for realistic whisker-array sequences and identify ConvRNN encoders, especially the IntersectionRNN variant, as superior to purely feedforward and state-space baselines for tactile categorization and neural alignment. The reported top-5 categorization figures are approximately $h_t$25 for a ResNet baseline, approximately $h_t$26 for S4, approximately $h_t$27 for ConvRNNs without an Attender, and approximately $h_t$28 for IntersectionRNN plus GPT Attender under supervised training. Neural alignment is measured by noise-corrected RSA Pearson’s $h_t$29, with ResNet at approximately $h_t$30, S4 at approximately $h_t$31, IntersectionRNN plus supervised training at approximately $h_t$32, IntersectionRNN plus SimCLR at approximately $h_t$33, and inter-animal consistency at $h_t$34. The paper further reports a Pearson correlation of $h_t$35 between supervised categorization accuracy and neural fit (Chung et al., 23 May 2025). This suggests that, at least in the reported tactile setting, nonlinear recurrent processing acts as a task-relevant and brain-aligned inductive bias.

Several unresolved questions recur across the literature. One is parallelism versus parameter efficiency: CRC and related recurrent factorizations reduce parameters, but sequential recurrent steps limit GPU or TPU parallelism (Retsinas et al., 2019). Another is stability versus expressivity: GRCL uses gates to prevent uncontrolled receptive-field expansion, while orthogonal or unitary convolutional recurrent operators are proposed to mitigate vanishing and exploding gradients without asymptotic overhead beyond FFT-based iteration (Wang et al., 2021, Magnasco, 2023). A third is architectural non-uniformity: some works call a model ConvRNN when convolution is recurrent, others when convolution and recurrence are staged, and still others when an RNN parameterizes the convolution filter itself (Hartmann, 2018, Zhao et al., 2018, Yang, 2018). The literature therefore supports a broad, family-level definition rather than a single standardized layer.

Future directions named explicitly in the cited work include replacing plain CRC recurrences with ConvGRU or ConvLSTM units, linearizing recursions to allow fuller parallelism, deploying recurrent channel-wise convolutions in detection and segmentation backbones, inserting attention or non-local modules over recurrent video features, and extending ConvRNN encoders with contrastive self-supervision and tactile-specific augmentations for label-free representation learning (Retsinas et al., 2019, Singh et al., 2018, Chung et al., 23 May 2025). The cumulative record indicates that ConvRNN research is best understood not as a narrow architectural template, but as a continuing effort to engineer locality, memory, causality, and efficiency within a single trainable framework.