URVC: A Unified Real-Time Neural Video Coding Model with Temporal, Spatial, and Perceptual Adaptivity

Abstract: Neural video coding has advanced rapidly, achieving competitive compression performance while also enabling real-time coding speed. Yet, existing codecs exhibit severe rigidity when deployed in dynamic environments, failing to adapt to different video content, user requirements, and quality preferences. First, to meet the real-time constraint, they discard explicit motion estimation and motion compression, thereby losing the ability to adapt temporal prediction to motion complexity and bitrate constraints. Second, their spatial bit allocation strategy is coarse and, once trained, is fixed. It cannot adapt to dynamic user requirements at test time, preventing users from freely controlling the spatial distribution of bits. Third, they cannot adapt their quality preference to varying application requirements without deploying separate models. We address all three limitations within a single real-time neural video codec--URVC, transforming a rigid system into a unified framework with temporal, spatial, and perceptual adaptivity. First, we propose a rate-aware adaptive temporal prediction method that generates diverse prediction candidates through a multi-candidate architecture and couples candidate selection directly to rate-distortion optimization. Second, we propose a decomposition-based spatial rate control method that achieves finer-grained spatial bit allocation through feature decomposition and separate quantization, and allows users to perform direct spatial rate control at test time without retraining. Third, we propose a perceptual switching method that only requires learning a secondary module bank alongside a frame generator, enabling a codec to switch between signal fidelity and perceptual quality modes.

• The paper introduces URVC, combining rate-aware adaptive temporal prediction, decomposition-based spatial rate control, and perceptual switching to restore test-time flexibility while maintaining real-time performance.
• URVC achieves a 22.4% average BD-rate reduction versus VTM-17.0 in PSNR mode, perceptual BD-rate gains of 57.3% for DISTS and 48.7% for LPIPS, and 20–30% bitrate savings for zero-shot ROI coding.
• The codec processes 1080p frames at 0.027 seconds for encoding and 0.028 seconds for decoding on an RTX 3090, demonstrating adaptive quality control at 223K MACs per pixel with 24.89 million parameters.

URVC is a real-time neural video codec that addresses a specific deficiency of efficient NVC designs: once trained, their coding strategies are fixed, so they cannot adapt temporal prediction to motion complexity and rate constraints, cannot honor user-specified spatial bit allocation at test time, and cannot switch between signal-fidelity and perceptual-quality objectives without deploying separate models. The authors build on DCVC-RT's implicit temporal modeling architecture (2607.15033) and add three mechanisms—rate-aware adaptive temporal prediction (RAATP), decomposition-based spatial rate control (DSRC), and perceptual switching—to restore these adaptive capabilities while retaining real-time throughput.

Motivation and problem statement

The paper's central diagnosis is that the efficiency gains of real-time neural codecs come from replacing explicit motion estimation/compression/compensation with a deterministic implicit prediction mapping $C_t = f_\theta(\hat{F}_{t-1})$, which implicitly assumes a unimodal Gaussian conditional distribution with constant variance. Under complex motion or low bitrate, this assumption fails, producing large residuals. Similarly, existing entropy-model quantization steps operate at latent resolution (16× downsampled), are fixed after training, and provide no user control; prior ROI codecs require ROI supervision during training and degrade non-ROI quality. Perceptual codecs (GAN-, VQ-, diffusion-, VAR-based) sacrifice signal fidelity even at high bitrates and remain computationally heavy—for example, DiffVC-RT consumes 2462K MACs/pixel—and typically require two separately trained models.

Rate-aware adaptive temporal prediction

RAATP replaces the single deterministic context predictor with three structurally heterogeneous branches: one full-resolution branch preserving fine detail, and two parallel branches operating on bilinearly downsampled inputs (enlarging receptive fields and suppressing high frequencies) with independent weights. This design is motivated by the Ambiguity Decomposition theorem from ensemble learning: fusing diverse estimators reduces total MSE through the diversity term.

Candidate fusion is performed by learnable per-channel quantization vectors $q_c^0, q_c^1, q_c^2$ indexed by QP within a module bank, followed by depth-wise convolutional fusion. Because these vectors act as differentiable proxies in the rate–distortion objective, their gradients jointly contain distortion-driven and rate-driven components scaled by $\lambda$: at low rates the optimization suppresses high-frequency channels of the full-resolution branch in favor of coarser candidates; at high rates fine detail contributes more fully. Rate-dependent mode selection thus emerges from end-to-end training without discrete search, restoring adaptivity lost in deterministic implicit prediction.

Decomposition-based spatial rate control

DSRC decomposes encoder and decoder intermediate features into a smooth structural component (obtained by down–up sampling) and a detail residual, each modulated by its own learnable quantization vector ($q_s$ for structure, $q_d$ for detail). Trained end-to-end, the network learns to heavily quantize structural content and preserve detail precision, yielding finer-grained spatial bit allocation than uniform channel modulation or latent-resolution entropy steps.

The key deployment property is that the down–up sampling decomposition can be replaced at test time by an arbitrary binary ROI mask: background pixels receive $q_s$ and foreground pixels receive $q_d$. Since $q_d$ has learned to preserve fine information and $q_s$ smooth content at lower fidelity, zero-shot ROI coding follows without retraining, additional supervision, or transmitted mask bits—the decoder side is unchanged.

Perceptual switching

The paper observes that the perception–distortion trade-off constrains only the realized conditional distribution $p_{\hat{X}|Y}$, not the architecture producing it. Since channel-wise quantization vectors merely rescale feature covariance as $q_c^0, q_c^1, q_c^2$0, switching quantization vectors changes the feature energy distribution without altering backbone convolution kernels. URVC therefore maintains two module banks (fidelity and perceptual, each containing all QP-indexed quantization vectors plus a factorized entropy model for $q_c^0, q_c^1, q_c^2$1) and two frame generators, sharing all convolutional weights. Training proceeds in two stages: joint fidelity training, then freezing the backbone and fine-tuning only the perceptual bank and frame generator under an LPIPS+GAN+MSE loss. Mode switching at inference costs negligible storage and identical computation, and the fidelity mode is provably unaffected because its parameters are frozen.

Experimental results

Trained on Vimeo-90k with variable-rate sampling over QP ∈ [0, 63] and hierarchical quality structure, and evaluated on HEVC Classes B–E, UVG, and MCL-JCV (YUV420, intra period −1):

Codec	Avg. BD-rate vs. VTM-17.0 (PSNR)
DCVC-FM	−25.2%
DCVC-RT	−18.8%
URVC	−22.4%

URVC surpasses the officially released DCVC-RT despite a constrained training protocol (256×384 crops instead of 512×512, grouped cascaded training with detached graphs due to GPU memory limits). Gains are largest on HEVC Class C (−25.4% vs. −17.4%) and Class D (−35.8% vs. −25.7%), though it remains slightly below DCVC-FM overall and slightly below released DCVC-RT at high bitrates on 1080p content—a limitation the authors attribute directly to the reduced training resolution and cascading scheme.

For perceptual quality, the perceptual mode achieves average BD-rates of −57.3% (DISTS) and −48.7% (LPIPS) against DCVC-FM, versus −7.0%/−4.9% for DCVC-RT. On BasketballDrive, the perceptual mode attains DISTS 0.098 / LPIPS 0.190 at 0.019 bpp, compared to 0.144 / 0.257 for DCVC-FM at 0.023 bpp.

For spatial rate control, evaluated on video motion control (compressing reference videos for character animation): applying DSRC yields 20–30% bitrate savings on reference videos while downstream generation metrics remain nearly unchanged (e.g., KungFu CLIP similarity 0.987 vs. 0.983, flow similarity 0.777 vs. 0.760, with 31.7% bitrate reduction).

Complexity remains firmly real-time: 0.027 s encode / 0.028 s decode per 1080p frame on an RTX 3090, at 223.00K MACs/pixel and 24.89M parameters—more than 20× faster than DCVC-DC/FM/PRAVC, with modest overhead over DCVC-RT (195.23K MACs, 19.93M params).

Ablations attribute −6.9% BD-rate to RAATP alone and a further −3.8% to DSRC (total −10.7% relative to the stripped baseline). For perceptual switching, module-bank switching alone yields −27.5% DISTS / −16.9% LPIPS BD-rate, rising to −51.8% / −44.1% when the perceptual frame generator is added, indicating both components contribute substantially.

Limitations and open questions

The paper concedes several constraints explicitly. The headline PSNR comparison against released DCVC-RT is confounded by unequal training conditions: the reproduced baseline DCVC-RT* reaches only −11.5%, so part of the reported gain over the official model may reflect architectural differences rather than the proposed methods alone, and the authors' own claim that larger-memory GPUs would yield further gains is unverified. High-bitrate performance on 1080p sequences trails the official DCVC-RT. The ROI evaluation is limited to two sequences and a single downstream generative task, leaving generalization to other ROI applications and mask granularities open. Perceptual switching supports exactly two modes; whether continuous interpolation between fidelity and perception points on the rate–distortion–perception surface is achievable with the same shared-backbone design remains unanswered. Finally, the binary-mask decomposition provides foreground/background granularity only; finer multi-region rate control is not demonstrated.

Conclusion

URVC demonstrates that the adaptivity traditionally provided by explicit mode decision in hybrid codecs can be recovered in a lightweight implicit-prediction neural codec through ensemble-style context prediction coupled to rate–distortion optimization, decomposition-based quantization that doubles as a test-time ROI mechanism, and module-bank switching that serves two quality objectives with one backbone. The combination of state-of-the-art real-time PSNR performance, large perceptual gains, and zero-shot ROI coding at roughly 223K MACs/pixel makes a concrete case that adaptivity and real-time efficiency need not be traded off in neural video coding.