CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation

Abstract: Implicit Neural Video Representation (INVR) has emerged as a novel approach for video representation and compression, using learnable grids and neural networks. Existing methods focus on developing new grid structures efficient for latent representation and neural network architectures with large representation capability, lacking the study on their roles in video representation. In this paper, the difference between INVR based on neural network and INVR based on grid is first investigated from the perspective of video information composition to specify their own advantages, i.e., neural network for general structure while grid for specific detail. Accordingly, an INVR based on mixed neural network and residual grid framework is proposed, where the neural network is used to represent the regular and structured information and the residual grid is used to represent the remaining irregular information in a video. A Coupled WarpRNN-based multi-scale motion representation and compensation module is specifically designed to explicitly represent the regular and structured information, thus terming our method as CWRNN-INVR. For the irregular information, a mixed residual grid is learned where the irregular appearance and motion information are represented together. The mixed residual grid can be combined with the coupled WarpRNN in a way that allows for network reuse. Experiments show that our method achieves the best reconstruction results compared with the existing methods, with an average PSNR of 33.73 dB on the UVG dataset under the 3M model and outperforms existing INVR methods in other downstream tasks. The code can be found at https://github.com/yiyang-sdu/CWRNN-INVR.git}{https://github.com/yiyang-sdu/CWRNN-INVR.git.

• The paper presents a novel framework that decomposes video representation into regular (network) and irregular (grid) components for improved spatiotemporal modeling.
• It introduces a coupled WarpRNN module to capture multi-scale temporal dynamics, achieving superior reconstruction with PSNR values up to 33.73 dB on benchmark datasets.
• The integration of a mixed residual grid enables efficient encoding of fine details and motion irregularities, resulting in substantial bitrate savings and high decoding speeds.

CWRNN-INVR: A Coupled WarpRNN-based Implicit Neural Video Representation

Introduction and Motivation

Implicit Neural Video Representation (INVR) leverages learnable neural networks and grid-based embeddings to encode video sequences, aiming for efficient storage, transmission, and downstream tasks. Prevailing INVR approaches focus exclusively on designing either neural network architectures or improved grid structures, often neglecting a systematic exploration of their complementary representational roles. The paper "CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation" (2604.06564) critically investigates this gap, demonstrating that neural networks excel at capturing structured, regular spatiotemporal patterns, while grids are more adept at representing irregular, frame-specific information. This dichotomy is grounded in both empirical reconstruction quality and theoretical information decomposition.

Figure 1: Performance distribution for different ratios of grid and network parameters, illustrating distinct representational strengths.

Figure 2: Residual maps highlighting the spatial and temporal irregularities captured by grids where neural networks fail.

Methodology

Mixed Network and Grid INVR Framework

The proposed CWRNN-INVR architecture implements a mixed model, explicitly decomposing video representation into regular (neural network) and irregular (grid) components. Each reconstructed frame $\hat{I}_t$ is a combination:

$\hat{I}_t = f_n(m)$

where $f_n$ is the spatial-temporal neural network and $m$ is the mixed residual grid. Both elements are jointly trained, enabling network reuse and streamlined encoding.

Coupled WarpRNN for Spatiotemporal Modeling

CWRNN-INVR introduces a Coupled WarpRNN module for explicit multi-scale temporal modeling. The module hierarchically processes local and global motion. Local features are separated via learned masks, processed by a WarpRNN to capture object movements; global features encapsulate background dynamics and are handled by a second WarpRNN. Motion alignment is performed by learned warping functions:

$$h_t = f_g(\tilde{x}_t, \mathrm{warp}(h_{t-1}, P_t(\tilde{x}_t)))$$

where $h_t$ is the hidden state, $P_t$ extracts motion features, and $\mathrm{warp}$ aligns spatially.

Figure 3: CWRNN-INVR framework overview and detailed Coupled WarpRNN module design.

Figure 4: Architecture of the WarpRNN module with a learned spatial warping function.

Figure 5: Visualization of learned initial hidden state, demonstrating anchor-like structural retention.

Mixed Residual Grid (MRG)

The MRG locally encodes appearance and motion irregularities. Temporal interpolation is used to obtain frame-specific residual features. These are fed back into the coupled WarpRNN, enhancing both local and global representation, allowing for efficient compensation of scene changes and object dynamics that neural networks alone struggle to capture.

Figure 6: R-D curves on UVG, revealing substantial improvements in rate-distortion efficiency.

Experimental Validation

Video Reconstruction

CWRNN-INVR outperforms prior state-of-the-art INVR models on UVG and DAVIS datasets, with the 3M parameter model achieving average PSNR values of 33.73 dB (UVG) and 31.22 dB (DAVIS), exceeding baseline models such as DSNeRV and HNeRV. Notably, the model preserves fine-frame details and handles both smooth and abrupt motion sequences exceptionally.

Figure 7: Qualitative reconstruction results on UVG and DAVIS (Jockey and Black swan), demonstrating preservation of motion details and texture.

Video Compression

The method delivers substantial bitrate savings: -54.24% BD-rate relative to HEVC and up to -23% improvement over competing INVR methods. Rate-distortion curves indicate superior performance across model sizes. Encoding time and GPU memory usage are comparable to prior INVR frameworks, with decoding speeds reaching 85.18 FPS, outperforming all neural and traditional codecs.

Ablation and Analysis

Model size ablation confirms scalability: even with 0.75M parameters, CWRNN-INVR outpaces NeRV and HNeRV. Coupled WarpRNN improves PSNR by nearly 1 dB for real videos versus single-layer RNNs. The addition of MRG consistently enhances spatial and temporal reconstruction, particularly for videos with substantial motion or irregular features.

Figure 8: Qualitative comparisons highlight the mixed residual grid’s ability to capture fine appearance details (e.g., hair, bee) unattainable by networks alone.

Implications and Future Directions

The explicit decomposition of video information in CWRNN-INVR enables efficient rate-distortion optimization and flexible adaptation to varying motion regimes. The coupled WarpRNN architecture advances temporal modeling in neural video compression. Practically, this framework sets a new benchmark for both storage and neural downstream tasks such as video editing or semantic querying.

Theoretically, this approach opens avenues for further exploration of joint neural-network/grid architectures, especially in dynamic scenarios with complex motions. Extensions may include hybridization with latent semantic embeddings or integration with neural radiance fields for volumetric video representation.

Conclusion

CWRNN-INVR establishes a rigorous framework for implicit neural video representation that leverages both neural networks for structured information and grids for irregular details. The coupled WarpRNN facilitates multi-scale temporal compensation, and the mixed residual grid efficiently encodes residual appearance and motion. The model achieves superior reconstruction, compression, and decoding results across benchmarks, validating the decomposition approach and offering a flexible foundation for future developments in neural video processing.