NVC-1B: Neural Video Coding at Scale

• The paper introduces NVC-1B, which, by scaling contextual modules, achieves unprecedented BD-rate savings and efficient video compression.
• It presents a conditional motion-compensated framework that combines a fixed motion estimator with deep CNN-based temporal context mining.
• Quantitative evaluations reveal significant PSNR and BD-rate improvements over baselines, underscoring optimal parameter allocation and training strategies.

NVC-1B is a neural video coding model exceeding one billion trainable parameters, specifically designed to address the application of large model scaling laws in neural video compression. By systematically increasing the parameter budgets of key architectural modules and empirically evaluating their impact, NVC-1B demonstrates substantial rate–distortion (RD) gains over prior approaches and sets a new benchmark in compression efficiency for end-to-end learned video coding frameworks (Sheng et al., 2024).

1. Model Architecture and Components

NVC-1B extends a conditional motion-compensated framework, with its design inheriting from the DCVC-SDD baseline. The network is composed of five principal coding modules and a fixed pre-trained motion estimator:

• Motion Estimation: Utilizes a frozen SpyNet to decompose each frame into “structure” and “detail” motion components, producing flows $v_t^s$ and $v_t^d$. This module is non-trainable (0.96 M parameters).
• Motion Encoder–Decoder (Inter): Compresses the flow pair $(v_t^s, v_t^d)$ via a lightweight CNN auto-encoder to obtain compressed latents $m_t$ and reconstructs quantized flows $(\hat{v}_t^s, \hat{v}_t^d)$. Contains 0.81 M parameters.
• Motion Entropy Model: Employs a hyper-encoder/decoder to model $m_t$ with quadtree spatial priors and Laplace distributions, governed by hyper-latents and spatial priors. Contains 2.50 M parameters.
• Temporal Context Mining (TCM): Builds a multi-scale feature pyramid from the previous decoder output $\hat{F}_{t-1}$, warps features using quantized motion, and aggregates long-term temporal information using a ConvLSTM. Produces three contextual tensors $C_t^0, C_t^1, C_t^2$. Major parameter allocation: 411.21 M.
• Contextual Encoder–Decoder (Rec): A large CNN auto-encoder that compresses current frame $x_t$; integrates temporal contexts at each scale to maximize temporal redundancy elimination. Parameter allocation: 504.59 M.
• Contextual Entropy Model: Hyperprior and spatial-prior-based compression with an additional temporal prior obtained from $C_t^2$. Parameter allocation: 435.88 M.

Total trainable parameters, excluding SpyNet: approximately 1.36 billion.

2. Optimization Objective and Training Procedure

The model is optimized using a Lagrangian rate–distortion objective:

$v_t^d$0

where:

• $v_t^d$1 is the expected rate for quantized motion latents and hyperpriors,
• $v_t^d$2 is the expected rate for contextual latents and their hyperpriors,
• $v_t^d$3 is distortion between decoded and original frames, being either MSE (for PSNR) or $v_t^d$4,
• $v_t^d$5 controls the RD tradeoff, chosen from $v_t^d$6.

For group-of-pictures (GOP) stability and to counteract error propagation, the multi-frame cascaded loss is used:

$v_t^d$7

Hierarchical training stabilizes the learning process by weighting P-frame contributions via a small scalar $v_t^d$8.

3. Scaling Analysis and Parameter Allocation

The parameter scaling process started from the 21 M parameter DCVC-SDD baseline. Modules were scaled individually to analyze their impact:

• Motion Encoder–Decoder: RD curve steeply improves with moderate scaling (to 52 M, BD-rate $v_t^d$915.4%) then regresses if oversized (at 96 M, $(v_t^s, v_t^d)$01.5%). Overparameterization impairs temporal fidelity and stability.
• Motion Entropy Model: Slight incremental BD-rate improvements (to $(v_t^s, v_t^d)$116.0%) with upscaling, but high capacities introduce instabilities.
• Contextual Encoder–Decoder: Monotonic BD-rate improvement ($(v_t^s, v_t^d)$210.6% at 50 M up to $(v_t^s, v_t^d)$314.8% at 90 M), signifying robust gains with upscaling.
• Contextual Entropy Model: Consistent, if modest, improvements, e.g., +0.8% BD-rate at 212 M.
• Temporal Context Mining: Alone, achieves up to $(v_t^s, v_t^d)$412.0% BD-rate on Class D. When combined with scaled contextual modules, overall BD-rate improves from $(v_t^s, v_t^d)$514.2% to $(v_t^s, v_t^d)$621.2%.

The final 1.36 B parameter allocation assigned approximately 90% of parameters to contextual AE, contextual entropy, and TCM, following empirical scaling laws.

4. Architecture Variants and Comparative Evaluation

Extensive ablation studies compared three macro-architectures for the contextual encoder/decoder, entropy, and TCM modules at matched parameter budgets:

• CNN-Only (Reference): Stacked $(v_t^s, v_t^d)$7 CNNs with residual connections delivered the best RD efficiency per parameter due to local receptive field advantages and strong spatial inductive bias.
• Mixed CNN+Transformer (Swin Layers): Swin–Transformer blocks replaced portions of residual blocks, evaluated at 42 M, 240 M, and 267 M. Gains noted over baseline, but pure CNN variants remained superior ($(v_t^s, v_t^d)$8 BD-rate at 240 M versus $(v_t^s, v_t^d)$9 for mixed).
• Transformer-Only (Swin Backbone): All convolutions and residuals replaced with Swin Transformer layers (44 M, 208 M, 232 M). Minor gains at lowest scale but large models rapidly collapsed (PSNR loss $m_t$04dB), highlighting poor stability and inferiority for fine quantization and spatial/temporal prior modeling.

The results indicate that hybrid and Transformer-heavy variants are suboptimal, at least for the motion-local and spatio-temporally structured tasks in neural video coding.

5. Quantitative Performance and Evaluation

BD-rate savings relative to VTM-13.2 (anchor codec) across test datasets are summarized as follows:

Dataset	DCVC-SDD (21 M)	NVC-1B (1.36 B)
HEVC B	$m_t$113.7%	$m_t$227.0%
HEVC C	$m_t$32.3%	$m_t$421.2%
HEVC D	$m_t$524.9%	$m_t$637.0%
HEVC E	$m_t$78.4%	$m_t$815.4%
UVG	$m_t$919.7%	$(\hat{v}_t^s, \hat{v}_t^d)$028.7%
MCL-JCV	$(\hat{v}_t^s, \hat{v}_t^d)$17.1%	$(\hat{v}_t^s, \hat{v}_t^d)$221.3%
Average	$(\hat{v}_t^s, \hat{v}_t^d)$312.7%	$(\hat{v}_t^s, \hat{v}_t^d)$425.1%

For YUV-PSNR, the average BD-rate saving is $(\hat{v}_t^s, \hat{v}_t^d)$520.5% versus $(\hat{v}_t^s, \hat{v}_t^d)$68.4% for DCVC-SDD. Inference time for 1080p frames is 4.44 s (encode) and 3.54 s (decode), a 5× cost over the baseline but markedly faster than classical codecs (e.g., VTM's encode time is 743.9 s).

6. Empirical Lessons and Architectural Implications

Empirical results establish several findings:

• Diminishing Returns in Motion Modules: Moderate upscaling is beneficial, but over-parameterization degrades sequential modeling and training dynamics.
• Strong Scaling for Contextual Modules: Contextual encoder, entropy, and TCM modules benefit consistently from larger parameterization.
• CNN Inductive Bias Predominance: Pure CNN architectures outperform Transformer-based options in perceptual quality and stability for video coding tasks due to their preferable local inductive biases.
• Energy Compaction and Warping: Larger models compact transform energy into fewer high-variance latent channels and enhance TCM feature warping accuracy.
• Resource and Stability Demands: Training >1B-parameter video codecs requires sophisticated scheduling, memory management, and stabilization techniques.

These observations suggest that “scaling laws” widely observed in NLP and vision transfer to neural video coding when module selection and architectural biases are respected.

7. Significance and Future Directions

NVC-1B demonstrates that strategically scaled neural architectures, especially when emphasizing the most “scaling-effective” modules, can yield state-of-the-art compression at practical runtimes (Sheng et al., 2024). Its results substantiate the extension of large model scaling laws to video compression, conditioned on deferred or minimal allocation of parameters to motion modules and the preservation of spatial inductive biases.

Plausible implications include further exploration of scaling behaviors in larger models and investigation into more memory- and compute-efficient design patterns for neural video coding at even greater parameter scales.