Masked Recurrent Networks

• Masked Recurrent Networks are a family of architectures that merge explicit recurrent state updates with diverse masking mechanisms to enforce prediction constraints.
• They employ various masking regimes—such as prediction masking, structural validity masking, and graph masking—to guide feature refinement and efficiency.
• These models are applied in video encoding, BEV perception, language modeling, and medical image segmentation, demonstrating improved accuracy and computational efficiency.

Searching arXiv for papers on “masked recurrent network” and closely related formulations. “Masked recurrent network” denotes a family of architectures that combine an explicit recurrent state update with a masking mechanism, but the phrase is not used in a single canonical sense across the literature. In contemporary usage, it can refer to a video encoder with dense token-level recurrence trained by asymmetric masked future-frame reconstruction, a BEV temporal fusion module whose recurrent output is filtered by a geometry-derived visibility mask, a 3D voxel refinement network driven by occupancy-aware masked GRU updates, or a hybrid system in which recurrence and masking are assigned to different subsystems such as recurrent multimodal alignment and masked graph fusion (Zoran et al., 15 Dec 2025, Yang et al., 2024, Wang et al., 23 Jul 2025, Meng et al., 2024). In masked language modeling, the same design space also includes bidirectional Transformer encoders that insert recurrence into the network while preserving the standard masked language modeling objective (Lei et al., 2022).

This suggests that the unifying criterion is not a particular cell type, loss, or masking operator. Rather, the common pattern is the conjunction of stateful propagation across steps and an explicit constraint, prior, or prediction target induced by masking.

1. Terminological scope and defining characteristics

The term spans several distinct technical meanings. In “Recurrent Video Masked Autoencoders,” “masked” refers to the self-supervised prediction target, while “recurrent” refers to a stateful token-latent update mechanism in the encoder; the model observes fully visible source frames, carries a latent state forward, and predicts a heavily masked future frame (Zoran et al., 15 Dec 2025). In “Geo-ConvGRU,” the recurrent component is a ConvGRU in BEV space, while masking is a geography-derived visibility prior applied to recurrently fused features (Yang et al., 2024). In “MonoMRN,” the paper uses the phrase literally: a Masked Recurrent Network built around a Masked Sparse Gated Recurrent Unit whose updates are restricted by a dynamically updated occupancy mask (Wang et al., 23 Jul 2025).

Other papers use the phrase more loosely. “Masked Graph Learning with Recurrent Alignment” combines a recurrent or iterative alignment module with a masked GCN fusion module; the recurrence and the masking do not inhabit the same operator, so the model is more accurately a hybrid of recurrent alignment and masked graph learning than a single monolithic masked RNN (Meng et al., 2024). “MRI lung lobe segmentation in pediatric cystic fibrosis patients” uses a recurrent MD-GRU backbone together with masked inputs or a whole-lung mask input channel, but not an internal learned masking mechanism inside the recurrent cell (Pusterla et al., 2021). Conversely, “State-Regularized Recurrent Neural Networks” constrains recurrence through a finite set of learnable states and stochastic transitions, yet does not apply a literal mask; it is adjacent to the topic through constrained recurrence rather than masking proper (Wang et al., 2019).

A recurrent misconception is to equate masked recurrent networks with either masked attention or classic masked RNNs for missing-data imputation. The surveyed work does not support that reduction. In these papers, “masked” may refer to masked prediction targets, graph-structural masking, occupancy masking, geographical validity masking, or binary ROI restriction, while “recurrent” may refer to sequence-time recurrence, token-position recurrence inside a Transformer layer, alignment-iteration recurrence, or spatial recurrence over a feature grid.

2. Canonical architectural patterns

A compact taxonomy of representative systems is given below.

Model	Recurrent component	Masking component
RVM	Transformer-based recurrent neural network over dense token grids	Random token-wise spatial masking on a future target frame
Geo-ConvGRU	ConvGRU over BEV features	Geometry-derived geographical mask over BEV cells
MonoMRN	MS-GRU with sparse 3D convolutions	Dynamically updated binary occupancy mask
MGLRA	LSTM contextual encoders plus MRFA/GRU refinement loop	Random graph masking in the masked GCN stage
SwishRNN MLM	Recurrent scan replacing each Transformer FFN	Standard BERT-style token masking
MD-GRU lung segmentation	Multi-dimensional recurrent segmentation backbone	Input-level lung masking or mask-channel conditioning

Two patterns dominate. The first is coupled masking and recurrence: the same module both carries state and is directly modulated by masking. RVM, Geo-ConvGRU, and MonoMRN belong to this category (Zoran et al., 15 Dec 2025, Yang et al., 2024, Wang et al., 23 Jul 2025). The second is factorized masking and recurrence: recurrence lives in one stage and masking in another, as in MGLRA’s recurrent alignment plus masked graph fusion, or the MRI lobe-segmentation workflow where recurrence is in MD-GRU but masking is imposed at the input or auxiliary-input level (Meng et al., 2024, Pusterla et al., 2021).

The architectural role of recurrence also varies sharply. RVM maintains a dense latent state of shape $(N+1)\times D$ aligned with patch tokens and a $[\mathrm{CLS}]$ token, making recurrence a token-grid memory rather than a clip-level summary (Zoran et al., 15 Dec 2025). Geo-ConvGRU and MonoMRN preserve spatial structure by replacing affine GRU operations with convolutions in BEV or voxel space (Yang et al., 2024, Wang et al., 23 Jul 2025). SwishRNN introduces recurrence across token positions inside each Transformer layer after self-attention, replacing the position-wise FFN rather than the attention block itself (Lei et al., 2022).

3. Masking mechanisms

Masking in this literature is heterogeneous and technically consequential. In RVM, masking is an asymmetric prediction protocol: source frames are fully visible, the future target frame is patch-tokenized, and $95\%$ of its tokens are randomly masked, with only the visible subset encoded by the shared ViT backbone. The visible target token set is written as

$$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$

with default mask ratio $m=0.95$; the future target is sampled with $\Delta t \in [4,48]$ frames after the last source frame (Zoran et al., 15 Dec 2025). In “Simple Recurrence Improves Masked LLMs,” masking remains standard BERT-style corruption: $15\%$ of input tokens are replaced by the special $[\mathrm{MASK}]$ token, while recurrence is inserted into the encoder architecture without changing the MLM objective (Lei et al., 2022).

A second masking regime is structural validity masking. Geo-ConvGRU defines a geographical mask $\mathcal{M}_{geo}$ over BEV voxels or cells using camera intrinsics, extrinsics, and the BEV projection geometry; valid cells receive value $1$, invalid ones a small positive constant $[\mathrm{CLS}]$0, with $[\mathrm{CLS}]$1 in experiments. The recurrently fused output is then masked as

$[\mathrm{CLS}]$2

so masking suppresses activations in geographically invalid or non-visible BEV cells (Yang et al., 2024). MonoMRN defines a binary occupancy mask

$[\mathrm{CLS}]$3

and applies that mask to both previous hidden state and projected 3D input before recurrent gate computation (Wang et al., 23 Jul 2025).

A third regime is graph masking. MGLRA describes random masking of the adjacency matrix and formalizes masked nodes and masked edges in the multimodal fusion graph. The paper repeatedly speaks of “random mask reconstruction on the nodes in the graph,” but the formalized training objective remains only the main classification loss; no explicit reconstruction decoder or reconstruction loss is supplied (Meng et al., 2024). A fourth regime is ROI restriction. In the MRI lung-lobe workflow, pseudo-MR and ufSSFP inputs are multiplied by a binary whole-lung mask to suppress anatomy outside the lungs, and one downstream network receives the whole-lung mask as an additional input channel rather than through internal recurrent gating (Pusterla et al., 2021).

4. Recurrent dynamics and computational implications

The recurrent operators themselves range from dense token-state recurrences to spatially structured GRUs. RVM defines its core abstractly as

$[\mathrm{CLS}]$4

where both $[\mathrm{CLS}]$5 and $[\mathrm{CLS}]$6 are dense token grids of shape $[\mathrm{CLS}]$7. Its gated update combines GRU-style gates with Transformer cross- and self-attention:

$[\mathrm{CLS}]$8

$[\mathrm{CLS}]$9

Because each step mixes the current frame’s $95\%$0 tokens with a state of size $95\%$1, unrolling over $95\%$2 frames yields linear temporal cost in $95\%$3, in contrast to full spatio-temporal attention over $95\%$4 tokens (Zoran et al., 15 Dec 2025).

Geo-ConvGRU uses standard ConvGRU equations in BEV space,

$95\%$5

$95\%$6

and then applies the geographical mask to the recurrent output. The masking is therefore post-recurrence rather than gate-level, which is a precise architectural distinction (Yang et al., 2024). MonoMRN moves the mask inside the recurrent update. Its MS-GRU computes

$95\%$7

$95\%$8

$95\%$9

$$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$0

so recurrence is simultaneously masked and sparse (Wang et al., 23 Jul 2025).

MGLRA contains three distinct sequential components: LSTM contextual encoders over utterance sequences, an MRFA loop recurrent over refinement iterations, and a GRU update

$$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$1

that captures “the correlation between feature alignments at each iteration.” Its most novel recurrence is therefore not sequence-time recurrence but recurrent cross-modal refinement (Meng et al., 2024). In the masked-language-model setting, SwishRNN performs a scan over token positions inside each encoder layer. With $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$2 and post-attention representations $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$3, it computes

$$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$4

$$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$5

followed by a gated output projection back to model dimension. The model remains bidirectional because the recurrent scan is placed after bidirectional self-attention (Lei et al., 2022).

5. Empirical behavior across application domains

In video representation learning, RVM is presented as evidence that dense token-level recurrence plus asymmetric masked prediction is a strong inductive bias. On large models, RVM-L $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$6 achieves $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$7 SSv2, $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$8 Kinetics, $$e^T \in \mathbb{R}^{(M+1)\times D}, \qquad M=\lfloor (1-m)N \rfloor,$$9 Waymo, $m=0.95$0 Perception Test, $m=0.95$1 ScanNet AbsRel, $m=0.95$2 DAVIS, $m=0.95$3 JHMDB, and $m=0.95$4 VIP, with normalized average $m=0.95$5, while RVM-H reaches normalized average $m=0.95$6. In the small-model regime, RVM-S at $m=0.95$7 parameters and without distillation scores $m=0.95$8 on SSv2, $m=0.95$9 on Kinetics, $\Delta t \in [4,48]$0 Waymo, $\Delta t \in [4,48]$1 Perception Test, $\Delta t \in [4,48]$2 AbsRel on ScanNet, $\Delta t \in [4,48]$3 on DAVIS, $\Delta t \in [4,48]$4 on JHMDB, and $\Delta t \in [4,48]$5 on VIP, with the authors claiming up to $\Delta t \in [4,48]$6 greater parameter efficiency on their averaged metric. On DAVIS videos longer than $\Delta t \in [4,48]$7 frames, RVM degrades substantially less than other video models and strong image baselines in label propagation over horizons of $\Delta t \in [4,48]$8, $\Delta t \in [4,48]$9, $15\%$0, $15\%$1, and $15\%$2 frames (Zoran et al., 15 Dec 2025).

In BEV perception, Geo-ConvGRU is positioned against 3D CNNs, ConvLSTM, and BEVFormer. In the temporal-module ablation at $15\%$3, ConvGRU attains long IoU $15\%$4 and long PQ $15\%$5, while Geo-ConvGRU improves to long IoU $15\%$6 and long PQ $15\%$7. At larger temporal fields, Geo-ConvGRU reaches long IoU $15\%$8 and long PQ $15\%$9 for $[\mathrm{MASK}]$0, and long IoU $[\mathrm{MASK}]$1 and long PQ $[\mathrm{MASK}]$2 for $[\mathrm{MASK}]$3. On NuScenes semantic segmentation, the reported IoUs are $[\mathrm{MASK}]$4, $[\mathrm{MASK}]$5, and $[\mathrm{MASK}]$6 under the three listed BEV settings; on perceived map prediction, the class-wise IoUs are Drivable area $[\mathrm{MASK}]$7, Lane $[\mathrm{MASK}]$8, Vehicle $[\mathrm{MASK}]$9, Pedestrian $\mathcal{M}_{geo}$0, Average $\mathcal{M}_{geo}$1; for future instance segmentation over a $\mathcal{M}_{geo}$2s horizon, the model reports future semantic IoU $\mathcal{M}_{geo}$3, PQ $\mathcal{M}_{geo}$4, SQ $\mathcal{M}_{geo}$5, and RQ $\mathcal{M}_{geo}$6 (Yang et al., 2024).

In monocular semantic scene completion, MonoMRN reports SC IoU $\mathcal{M}_{geo}$7, SSC mIoU $\mathcal{M}_{geo}$8, and FPS $\mathcal{M}_{geo}$9 on NYUv2, and SC IoU $1$0, SSC mIoU $1$1 on SemanticKITTI. The ablation isolates the recurrent masked design: a standard GRU yields SC-IoU $1$2, SSC-mIoU $1$3, Params $1$4M, MACs $1$5G, whereas MS-GRU yields SC-IoU $1$6, SSC-mIoU $1$7, Params $1$8M, MACs $1$9G. The full component study moves from Baseline $[\mathrm{CLS}]$00 to $[\mathrm{CLS}]$01MS-GRU $[\mathrm{CLS}]$02, then $[\mathrm{CLS}]$03DAP $[\mathrm{CLS}]$04, and finally $[\mathrm{CLS}]$05Mask Updating $[\mathrm{CLS}]$06 (Wang et al., 23 Jul 2025).

In multimodal emotion recognition in conversation, MGLRA achieves $[\mathrm{CLS}]$07 accuracy and $[\mathrm{CLS}]$08 weighted F1 on IEMOCAP, and $[\mathrm{CLS}]$09 accuracy and $[\mathrm{CLS}]$10 weighted F1 on MELD. The ablation on IEMOCAP shows a progression from $[\mathrm{CLS}]$11 to $[\mathrm{CLS}]$12 with graph attention filtering, to $[\mathrm{CLS}]$13 with MRFA, to $[\mathrm{CLS}]$14 with multi-head attention, and to $[\mathrm{CLS}]$15 with masked GCN. The efficiency comparison reports that MGLRA without mask uses $[\mathrm{CLS}]$16 MB and $[\mathrm{CLS}]$17 s with $[\mathrm{CLS}]$18 performance, whereas MGLRA with mask uses $[\mathrm{CLS}]$19 MB and $[\mathrm{CLS}]$20 s with $[\mathrm{CLS}]$21 (Meng et al., 2024).

In masked language modeling, replacing every FFN with SwishRNN raises the average score across $[\mathrm{CLS}]$22 GLUE/SuperGLUE tasks from $[\mathrm{CLS}]$23 for BERT-orig to $[\mathrm{CLS}]$24 in the base model, an absolute improvement of $[\mathrm{CLS}]$25 points, and from $[\mathrm{CLS}]$26 to $[\mathrm{CLS}]$27 in the large model, a gain of $[\mathrm{CLS}]$28 points. The recurrent model is also reported to reach lower MLM pretraining loss and to be more stable in fine-tuning across learning rates (Lei et al., 2022). In medical image segmentation, the best lung-lobe model, Network-2, obtains an average all-lobe Dice similarity coefficient of $[\mathrm{CLS}]$29 and an average median Hausdorff distance of $[\mathrm{CLS}]$30 mm, outperforming Network-1 and Network-3 in both accuracy and robustness (Pusterla et al., 2021).

6. Related concepts, limitations, and open problems

The category boundary remains technically important. A masked recurrent network is not necessarily a recurrent network with masked hidden states. In RVM, the masked object is the prediction target; in Geo-ConvGRU, the mask is a deterministic visibility prior applied to recurrent output features; in MonoMRN, the mask directly restricts recurrent computation to likely occupied voxels; in MGLRA, recurrence and masking belong to different modules; and in the MRI workflow, masking primarily performs ROI restriction and domain bridging rather than recurrent modulation (Zoran et al., 15 Dec 2025, Yang et al., 2024, Wang et al., 23 Jul 2025, Meng et al., 2024, Pusterla et al., 2021).

A closely related but distinct line of work constrains recurrence without masking. State-Regularized RNNs introduce a finite set of learnable centroids $[\mathrm{CLS}]$31 and a probability distribution over them, with practical hidden-state update

$[\mathrm{CLS}]$32

This imposes a structured transition bottleneck and makes the model behave like a probabilistic finite automaton or, as $[\mathrm{CLS}]$33, like a deterministic finite automaton for memoryless recurrent cells, but it does not apply a literal mask (Wang et al., 2019). This suggests that the broader research theme is structured recurrence, of which masking is one realization.

The limitations are correspondingly diverse. RVM notes that sequential frame processing can be computationally heavier than tubelet-based short-clip transformers for very short sequences, and that training is memory-intensive because backpropagation through time must retain activations from the ViT encoder at each recurrent step; the paper also reports no saturation point in scaling (Zoran et al., 15 Dec 2025). Geo-ConvGRU relies on known camera intrinsics and extrinsics, uses a binary-like mask with values $[\mathrm{CLS}]$34 or $[\mathrm{CLS}]$35, applies the mask after rather than inside the recurrent gates, and leaves the exact default temporal field somewhat ambiguous across sections (Yang et al., 2024). MonoMRN reports that performance improves from $[\mathrm{CLS}]$36 to $[\mathrm{CLS}]$37 to $[\mathrm{CLS}]$38 iterations but degrades at $[\mathrm{CLS}]$39, so recurrent refinement is beneficial but not unbounded (Wang et al., 23 Jul 2025). MGLRA uses “random mask reconstruction” language without specifying an explicit reconstruction loss, and omits full hidden-dimensional and GCN-layer detail (Meng et al., 2024). SwishRNN remains slower than a pure Transformer, with naive scan() about $[\mathrm{CLS}]$40 slower and variable step sizes reducing the slowdown to about $[\mathrm{CLS}]$41–$[\mathrm{CLS}]$42 on TPUs (Lei et al., 2022). The MRI segmentation workflow, while empirically effective, does not provide optimizer, learning-rate, batch-size, or explicit loss details in the reproduced description (Pusterla et al., 2021).

A plausible implication is that future work will continue to separate into at least three trajectories: masked predictive recurrence, where masking defines the training target; masked state propagation, where recurrence itself is spatially or structurally filtered; and hybrid pipelines, where recurrent reasoning and masking are allocated to different stages. The surveyed literature indicates that all three are viable, but that they should not be conflated under a single narrow definition of the term.