- The paper introduces a robust, cross-platform learned video codec that overcomes floating-point inconsistencies via novel codebook-indexed entropy parameter transmission.
- It integrates a unified I/P-frame model, gated memory modules, and hardware-friendly activations to balance rate-distortion efficiency with practical real-time deployment.
- Experimental results demonstrate over 70% BD-rate improvement and error-free performance across diverse NPUs, marking a breakthrough in learned video coding.
Introduction
The MLVC codec proposes a comprehensive solution to one of the most pressing challenges for learned video compression: robust, efficient, and cross-platform deployment on commodity neural processing units (NPUs). While neural codecs substantially outperform traditional video codecs (e.g., H.265/HEVC) in rate-distortion trade-offs, their deployment has been impeded by catastrophic cross-platform failures primarily due to inconsistent floating-point arithmetic and entropy coding divergences. MLVC introduces a novel architecture and training methodology that resolves this failure mode while preserving the competitive coding efficiency, subjective quality, and real-time speed required by practical use cases such as video conferencing.
Technical Innovations
The core advance in MLVC is explicit codebook-based transmission of scale parameters involved in entropy modeling. This approach leverages two insights: (a) spatial and inter-channel correlation among scale parameters allows for aggressive parameter sharing and substantial overhead amortization, and (b) the parameter sharing and transmission can be efficiently entropy-coded in the hyperprior latent stream, eliminating the dependency on bit-exact adaptive prior replication at the decoder. This design choice ensures that decoder and encoder use identical entropy model distributions, forcibly eliminating the catastrophic symbol stream divergence observed in state-of-the-art neural codecs (e.g., DCVC-RT).
The mechanism is formalized as follows:
- Scale indices are computed as quantized and spatially/channel-shared mappings from the hyperlatents.
- During decoding, these indices deterministically generate the scale parameters by lookup from a fixed codebook, ensuring all platforms reconstruct precisely the same entropy model parameters, regardless of floating-point quirks.
Architectural Modifications for Efficient Real-World Deployment
MLVC makes several modifications to baseline neural video codec architectures to recover rate-distortion performance lost to the explicit transmission overhead and ensure NPU compatibility:
- Unified I-frame/P-frame Model: By representing I-frames as P-frames with a constant gray reference, model size and deployment complexity are reduced, at the cost of a modest BD-rate penalty.
- Gated Memory Module: Inspired by LSTM and recent advances in memory-augmented video models, the MLVC decoder incorporates a temporal memory gating mechanism. This provides increased capacity for long-term temporal dependencies with minimal overhead (tripling only final 1×1 conv outputs), which is critical for occlusion recovery and cross-frame consistency.
- Hardware-Compatible Activations (ReGLU, ReLU/LeakyReLU): Cross-platform divergence is further suppressed by constraining nonlinearity implementations to simple, piecewise-linear activations (ReLU, LeakyReLU). For improved expressivity, MLVC proposes a capped gated activation (ReGLU), which enables controlled gating without incurring NPU kernel divergence.
- Long-Term Reference (LTR) Recovery: To control prediction-chain drift caused by residual floating-point differences, periodic LTR frames are inserted. This mechanism, akin to intra-refresh and reference recovery in classical codecs, substantially lengthens error-free inter-frame chains without the high cost of frequent I-frames.
- Perceptual Losses with ROI Weighting: To close the gap between architecture-constrained (cross-platform) and unconstrained learned codecs, training is finetuned using perceptual losses (e.g., LPIPS) and region-of-interest weighting based on task-specific priorities (e.g., faces for conferencing).
Experimental Results
Rate-Distortion Efficiency and Subjective Quality
MLVC's subjective and objective performance is demonstrated on the VCD video conferencing dataset and established HEVC test sets. It achieves over 70% BD-rate (MOS) improvement relative to hardware HEVC baselines (HEVC-QSV) at standard video conferencing resolutions, and subjective quality competitive with unconstrained, state-of-the-art research codecs such as DCVC-RT. The entire coding pipeline, including both encoder and decoder, sustains real-time throughput (>100 FPS at 360p; 30 FPS at 1080p for the MLVC-S variant) on NPUs from Apple, Intel, and Qualcomm.
A robust suite of cross-device evaluations (encoding and decoding on divergent platforms) confirms zero catastrophic failures and only minor (<2 BD-rate point) average quality fluctuations across the tested hardware matrix. Notably, DCVC-RT and other prior neural codecs fail catastrophically under these conditions (with infinite BD-rate cost).
Ablation and Scalability
Ablation studies characterize the incremental effect of each design adaptation for cross-platform compatibility. The main findings are:
- The scale-sharing mechanism incurs a BD-rate penalty (~6-18 points vs. unconstrained models), almost fully reclaimed via memory/gating/loss optimizations.
- Hardware-friendly activations (ReLU/LeakyReLU/ReGLU) are the only class that ensures error-free inference across all major NPU platforms.
- LTR frames efficiently trade off between minimizing error drift and intra-frame rate cost, and their periodicity can be tuned according to divergence characteristics of emerging hardware.
Benchmarks on larger models suggest consistent improvements in rate-distortion efficiency as NPU capabilities scale, highlighting the architecture's extensibility.
Implications and Future Directions
The MLVC codec represents a decisive step towards operationalizing learned video compression for latency- and reliability-critical consumer applications. Its main technical implication is that entropy model determinism—when achieved without reliance on bit-exact low-level arithmetic—permits learned codecs to be as robust as classical codecs in practical, heterogeneous deployments. The architectural approach adopted here (codebook-indexed entropy parameter transmission, memory modules, and minimal activation divergence) generalizes to other sequence and temporal modeling domains that require tightly bound cross-device inference determinism.
Practically, the real-time, high-quality, and platform-robust operation of MLVC makes it viable for integration into large-scale video conferencing and streaming systems—domains from which learned codecs have traditionally been excluded.
The main open challenge is power efficiency. As real-time inference is now possible on commodity NPUs at competitive quality, the focus will likely shift to reducing energy consumption per encoded bit and supporting simultaneous encode/decode workloads at high resolutions. Additionally, as NPU hardware and kernel implementations evolve, standards for arithmetic determinism and further optimization of cross-platform quantization flows will become increasingly relevant.
Conclusion
MLVC provides the first learned video codec to robustly close the cross-platform reliability gap while maintaining real-time operation and strong coding efficiency. By combining entropy model determinism via explicit parameter transmission, hardware-friendly and expressive architectural choices, and domain-optimized perceptual training, MLVC establishes a new deployment-ready baseline for neural codecs. Its demonstrated results enable, for the first time, the practical adoption of learned video coding in applications where cross-device compatibility is non-negotiable. Future work should consider detailed energy/latency trade-offs, support for higher resolutions and bi-directional prediction, and broader standardization of hardware runtime behavior.