Ultra-Fast Neural Video Compression
Abstract: While neural video codecs (NVCs) have demonstrated superior compression ratio, their prohibitive computational complexity remains a critical barrier to real-world deployment. This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off. Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously. This is enabled by cross-frame interaction modules for joint spatial-temporal modeling and frame-specific decoders for parallel reconstruction. This paradigm not only dramatically enhances coding throughput but also facilitates more effective modeling of long-term temporal correlations. To further boost speed, we propose a streamlined entropy coding mechanism that consolidates bit-stream interactions into a single step, substantially reducing decoding overhead. Building on these innovations, we present DCVC-UF (Ultra-Fast), a new NVC that sets a new SOTA in performance. Our experiments show that DCVC-UF can achieve ultra-fast encoding and decoding speeds, significantly outperforming previous leading codecs. DCVC-UF serves as a notable landmark in the journey of NVC evolution. The code is at https://github.com/microsoft/DCVC.
Summary
- The paper introduces DCVC-UF, a neural video codec that replaces explicit motion coding with a chunk-based framework for efficient encoding and decoding.
- It employs parallel chunk processing and a single-step entropy model, achieving up to 42.2% bitrate savings and over 3× speedup compared to previous methods.
- The design scales effectively with modern GPUs, enabling long-sequence training and real-time, high-resolution video compression for practical deployment.
Ultra-Fast Neural Video Compression: A Technical Summary
Introduction
Recent research in neural video codecs (NVCs) emphasizes not only maximizing compression efficiency (rate-distortion performance), but also drastically reducing system complexity to enable practical deployment. The paper "Ultra-Fast Neural Video Compression" (2606.04410) introduces DCVC-UF, a neural codec that innovates on both the architectural and algorithmic level. It leverages chunk-based coding with joint spatial-temporal modeling and employs a parallel, highly optimized entropy coding mechanism, realizing unprecedented encoding/decoding throughput with state-of-the-art compression. The design exhibits significant improvements over both conventional hierarchical-B-based codecs and prior NVC approaches with notably higher speed and competitive or superior rate-distortion-complexity trade-off.
Chunk-Based Coding Paradigm
Conventional neural and traditional codecs process video in a sequential frame-by-frame format, with options for hierarchical temporal referencing (e.g., hierarchical-B frames), but they are inherently limited by sequential dependencies and costly motion estimation/compensation steps. The DCVC-UF framework ditches explicit motion coding entirely and introduces a chunk-based architecture: input videos are segmented into non-overlapping chunks (e.g., 8 frames each), which are encoded simultaneously into a single latent representation and reconstructed in parallel with dedicated per-frame decoders. This enables efficient spatial-temporal modeling and directly leverages architectural parallelism.
Figure 1: Coding paradigm comparison—hierarchical-B (frame-wise, motion-dependent) vs. chunk-based (parallel, motion-free), demonstrating the elimination of explicit motion and improved throughput.
The model applies cross-frame interaction modules to capture jointly the spatial and temporal dependencies within each chunk, and frame-specific decoders specialize to reconstruct each frame optimally. This explicit specialization (reminiscent of MoE designs) leads to more robust adaptation to diverse video structure and motion patterns.
Figure 2: Schematic of the DCVC-UF pipeline: chunk input is patchified, processed by cross-frame encoders, quantized, and decoded in parallel by frame-specific experts.
Streamlined Entropy Coding
A significant computational bottleneck in NVCs is entropy coding/decoding, especially when using autoregressive or even multistep context models. DCVC-UF pioneers a streamlined entropy coding mechanism: the latent produced by the chunk encoder is partitioned via a quadtree structure, but the scale parameters (standard deviation of the assumed Gaussian prior for each partition) are estimated for all partitions in a single network invocation, thereby collapsing bitstream read operations into a single, highly parallel step. In contrast to previous methods where both the scale and mean need iterative/serial estimation, this change minimizes memory I/O and synchronization overhead, maximizing throughput and device utilization.
Figure 3: (a) Quadtree partitioning of latent; (b) previous multi-step decoding with interleaved bitstream/parameter estimation; (c) DCVC-UF’s single-step scale estimation for all partitions, sharply accelerating decoding.
Modeling Long-Term Temporal Correlation
Training NVCs to exploit long-range temporal redundancy generally confronts the scalability bottleneck of storing and modeling separate latents for each frame. The chunk-based scheme in DCVC-UF encodes the entire chunk into a compact latent, drastically reducing memory footprint and allowing effective training on much longer sequences (up to 1024 frames at in 24GB GPU, when batch size is one). This approach exposes the model to longer temporal context, enhancing its ability to learn implicit and persistent temporal structure, which empirically improves compression ratios.
Empirical Results
DCVC-UF establishes new SOTA benchmarks for rate-distortion-complexity on multiple datasets (UVG, HEVC, MCL-JCV). For example, with chunk size 8 and the "large" configuration (DCVC-UF HT-L), the model achieves a 42.2% average bitrate saving over VTM (LD), with encoding and decoding speeds of 371.1 FPS and 273.6 FPS, respectively, on 1080p content with an RTX 4090. The small model (HT-S) achieves 31.6% bitrate savings with even higher throughput. The speedup over previous neural codecs is more than 3×, and DCVC-UF can operate in both high-throughput (multi-frame) and low-delay (single-frame) regimes.
Figure 4: Encoding/decoding FPS versus model type and hardware, illustrating sharp improvements and scalability with newer GPUs.
Figure 5: Rate-distortion (PSNR vs. BPP) curves on UVG show DCVC-UF (HT-L/HT-S) outperforms prior NVCs, particularly at moderate-to-low quality (higher compression).
The chunk-based paradigm scales favorably with general-purpose GPUs, benefiting from ongoing advances in AI hardware without algorithmic modification. The method’s real-time encoding/decoding exceeds 1,400 FPS on datacenter GPUs (B200) for 1080p content.
Ablation and Architectural Choices
Ablation studies confirm distinct gains from each architectural innovation:
- Chunk coding alone increases speed but may degrade compression without per-frame specialization.
- Frame-specific decoders restore and surpass original rate-distortion efficiency.
- Streamlined entropy model accelerates decoding by 30%+ with negligible loss.
- Long-sequence training (enabled only by chunk-latent representation) provides substantial compression gains at fixed speed.
Theoretical and Practical Implications
This work demonstrates the practical viability of NVCs for large-scale, real-time video applications, previously hampered by NVCs’ complexity. The results suggest that explicit motion modeling may become obsolete in future high-throughput codecs, replaced with joint spatiotemporal transformers/encoders that operate over frame chunks. The parallel chunk decoding aligns naturally with contemporary compute platforms, minimizing device synchronization and opening opportunities for efficient deployment in video streaming, cloud analytics, and adaptive-rate controls.
Practically, DCVC-UF's scalable design positions it for cloud and edge scenarios, where device-agnostic, high-throughput compression is required. The architecture is architecturally simple (built almost entirely from depthwise-convolution blocks), facilitating further extension. Future work may investigate adaptive chunk sizing for content-dependent throughput/latency control and enhanced transformers as backbone components.
Conclusion
DCVC-UF presents a robust, architecturally streamlined approach to neural video coding, coupling chunk-based, motion-free coding with specialized decoders and single-step entropy coding, leading to the fastest and among the most efficient NVCs reported to date. The implications for further gains in complexity reduction and practical deployment are substantial, and the design offers a template for future work on adaptive, content-aware NVC architectures (2606.04410).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Ultra-Fast Neural Video Compression — Explained Simply
What is this paper about?
This paper introduces a new way to shrink (compress) videos so they take up less space and can be sent or played much faster, without making them look worse. The method is called DCVC-UF (Ultra-Fast). It uses AI to compress several video frames together at once, which makes it both faster and better at finding patterns over time.
What questions are the researchers trying to answer?
- How can we make AI video compression not just high-quality and small in size, but also extremely fast?
- Can we avoid the slow, step-by-step parts that current AI video compressors use?
- Is there a smarter way to use information across many frames, so we save more bits while keeping the video looking good?
How does their method work? (In everyday language)
First, a quick idea: a video is a sequence of pictures (frames). Traditional AI methods compress one frame after another and spend a lot of time estimating how things move between frames. That’s slow and complicated.
This paper flips that idea:
- Think “chunks,” not single frames:
- Instead of handling frames one by one, the video is split into small groups (chunks), like 8 frames at a time.
- The model turns this 8-frame chunk into one compact “summary” (a small bundle of numbers called a latent). Then it decodes all 8 frames together.
- Analogy: Instead of summarizing each page of a comic separately, you write one summary for a whole chapter, then use it to recreate all the pages at once.
- Cross-frame teamwork:
- The model has special modules that let frames “talk to” each other, so it can learn how things change over time without explicitly tracking every motion.
- Analogy: It’s like having a group discussion to understand a scene, not interviewing each person one-by-one.
- Frame-specific decoders:
- Even though the chunk is summarized together, each position in the chunk (like frame 1, frame 2, … frame 8) gets its own small “specialist” decoder.
- Analogy: If you have 8 translators, each one specializes in the kind of sentences that usually appear in a certain part of the chapter. This makes the final reconstruction sharper and more accurate.
- Learning long-range patterns:
- Because one compact summary represents many frames, the system can be trained on longer videos without running out of memory. That helps it learn long-term trends (like how a person walks across a room over several seconds), which improves compression.
- Faster “bitstream” decoding (streamlined entropy coding):
- When turning the compact summary into actual bits for storage, older methods do multiple back-and-forth steps with the “bitstream” (like unzipping parts in sequence).
- The new trick: separate two kinds of information (“scale” and “mean”). The scale tells how uncertain each value is and is the only part needed to decode the bits. The mean can be added after.
- By predicting all the scales in one shot, the system can decode all parts in a single pass, cutting a lot of overhead.
- Analogy: Instead of asking for permission four times to open four locks, you get one master key and open them all at once.
What did they find, and why is it important?
- It’s very fast:
- On a strong GPU, DCVC-UF can encode and decode 1080p videos at hundreds of frames per second (FPS). One version hits around 656 FPS for encoding and 453 FPS for decoding; on a bigger data center GPU, it reaches over 1400 FPS encoding.
- This is much faster than many previous AI video codecs.
- It compresses well:
- Compared to a top traditional standard (H.266/VTM), it can save around 31–42% of the bits at similar quality in the “high-throughput” mode (when using chunks like 8 frames). That means smaller files or less bandwidth needed for streaming.
- There’s also a low-delay version (chunk size = 1) for real-time uses that still runs very fast and saves bits, though the compression is less dramatic than the chunked version.
- It scales with future hardware:
- Because it’s built with standard GPU operations, as GPUs get better, this codec naturally gets faster without special hardware changes.
In simple terms: it makes videos smaller and faster to process, and it keeps looking good. That’s great for streaming, storage, and real-time video apps.
What impact could this have?
- Smoother streaming and gaming: Faster compression and decompression help reduce delays and buffering.
- Cheaper storage and bandwidth: Smaller video sizes mean less data to store and send.
- Ready for the future: As AI chips get better, this method automatically speeds up.
- Practical for many uses: From video calls and live events (low delay) to video platforms and cloud services (high throughput).
A note on limitations: Right now, the chunk size is fixed (for example, 8 frames). In the future, the system could learn to vary chunk size depending on how busy or calm the video is, possibly improving both speed and quality even more.
Key takeaways
- Compress multiple frames together (chunks), not one by one.
- Use specialized decoders for each frame position to get better quality.
- Decode bits in one go by predicting the “scale” information up front.
- Train on longer videos to learn long-range motion and patterns.
- Achieve record-breaking speed while saving a lot of data at the same quality.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, consolidated list of concrete gaps and unanswered questions that future work could address:
- Adaptive chunking: The codec uses a fixed chunk size (e.g., N=8); no method is provided to adapt chunk size or boundaries to content dynamics, latency budgets, or device constraints.
- Chunk size sweep: There is no systematic study of how chunk size affects the rate–distortion–latency–throughput trade-off across diverse content (e.g., low/high motion, scene complexity, frame rate).
- Random access and scene cuts: Intra-period is set to −1 and random-access strategy is unspecified; handling of scene cuts, frequent shot changes, and intra refresh (IDR/CRA) is not evaluated or designed.
- Error resilience: Robustness to packet loss, bitstream corruption, or drift (especially given chunk-level dependencies and propagated context) is untested; no error concealment or resynchronization mechanisms are proposed.
- First-chunk cold start: The coding strategy and overhead for the first chunk (without prior temporal context) are not described or quantified.
- Per-frame bit allocation within a chunk: A single latent is shared across all frames in a chunk, limiting per-frame rate control; no mechanism for uneven complexity allocation across frames (or regions) within a chunk is provided.
- Frame-specific decoders’ scalability: Model size scales with the number of frames per chunk; there is no solution for variable N, large N, or parameter-efficient sharing (e.g., conditional modulation/MoE routing) without linear parameter growth.
- Generalization to variable chunk sizes at test time: It is unclear whether models trained at a fixed N can decode different N at inference without retraining.
- Chunk boundary artifacts: Potential artifacts or temporal inconsistencies at chunk boundaries (and whether overlapping/sliding windows would help) are not analyzed.
- Long-term drift: While features are propagated chunk-to-chunk, the paper does not quantify drift accumulation over very long videos or propose drift mitigation/reset strategies.
- Entropy model trade-offs: The streamlined entropy model’s RD penalty versus fully coupled models is only partially ablated; alternative designs (e.g., partial coupling, hierarchical/mixture priors, normalizing flows) and their speed/RD trade-offs are not explored.
- Fully GPU-native bitstream pipeline: The arithmetic coder’s device placement and full GPU implementation are not specified; a design and evaluation of a GPU-native AE/AD path (and PCIe synchronization costs) is missing.
- Pipeline parallelism: Cross-chunk pipeline parallelism (overlapping network inference and entropy coding) is acknowledged but not implemented or quantified; end-to-end latency under pipelining is unreported.
- Rate control for streaming: There is no VBV/HRD-constrained rate control, CBR/ABR policy, or chunk-level bitrate smoothing strategy, which are required for practical streaming.
- Perceptual quality: Evaluations focus on YUV420 PSNR; perceptual metrics (MS-SSIM, VMAF), temporal stability (flicker), and subjective tests are absent, especially at low bitrates.
- Content coverage: No analysis across HDR/10-bit, wide color gamut, film grain, gaming/animation, high-FPS content, or 4:4:4 chroma; applicability beyond YUV420 8-bit remains untested.
- UHD RD performance: 4K/8K throughput is reported, but rate–distortion performance at UHD is not presented; scalability of RD with resolution is unknown.
- Memory footprint: VRAM usage and memory-bandwidth pressure at inference (vs. resolution, batch size, chunk size, and model scale) are not measured; this is critical for deployment.
- Energy efficiency: No measurements of power or energy-per-frame across devices; comparisons against traditional codecs and prior NVCs on energy efficiency are missing.
- Device portability: CPU-only performance, mobile/edge NPUs, and non-NVIDIA GPU backends are not evaluated; portability constraints and kernel dependencies are unclear.
- Data for long contexts: Although training can theoretically use 512–1024-frame sequences, fine-tuning used 128-frame data due to dataset scarcity; the actual gains from 256/512/1024-frame training remain unquantified.
- High-motion/occlusion robustness: Failure cases and performance on extremely fast motion, large occlusions, and nonrigid dynamics are not analyzed; explicit mechanisms for such content are absent.
- High-quality regime: The approach trails VTM (Hierarchical-B) above ~40 dB; techniques to close the gap at near-lossless quality (e.g., multi-scale latents, refined decoders) are not explored.
- Standardization and bitstream syntax: There is no discussion of a stable bitstream format, versioning, random access points, or interoperability with existing standards.
- Comparisons beyond DCVC series: Head-to-head comparisons against recent non-DCVC learned codecs (including generative/INR/Gaussian approaches) under matched encoding budgets are limited or absent.
- Temporal irregularities: Support for variable frame rates, dropped/inserted frames, and non-uniform time steps within chunks is not addressed.
- Scalability layers: Temporal/spatial/scalable enhancement layers are not supported; chunk-based coding for scalable streaming and adaptive playback remains open.
- Rate–compute adaptivity: Online adaptation of model complexity (e.g., gating experts, changing decoder depth) to meet device or battery constraints is not investigated.
- Training objectives: Loss design (rate–distortion weighting, temporal/perceptual terms), lambda schedules, and their impact on perceptual vs. PSNR outcomes are not detailed.
Practical Applications
Below is an overview of practical, real-world applications enabled by the paper’s findings and innovations—chiefly, chunk-based neural video coding, frame-specific decoders, and a streamlined entropy model that together deliver state-of-the-art rate–distortion–complexity and unprecedented GPU throughput.
Immediate Applications
These can be deployed now in controlled software ecosystems that can ship both encoder and decoder, particularly on GPU-equipped servers and workstations.
- High-throughput VOD transcoding and storage optimization
- Sectors: media/entertainment, CDN, cloud storage.
- What to do: Use DCVC-UF (HT-L for maximum bitrate savings, HT-S for maximum throughput) to transcode catalogs and mezzanine assets; reduce storage and egress bandwidth for 1080p–4K libraries with lower delay than hierarchical-B.
- Tools/workflows: Integrate the open-source DCVC repository into FFmpeg pipelines; run on GPU instances (e.g., A100/H100/L40S/B200); containerize in Kubernetes for batch transcodes.
- Assumptions/dependencies: Decoder must be available wherever playback occurs (non-standard codec). Best suited to closed ecosystems (first-party apps, OTT apps with embedded decoders) or internal archives. Requires GPUs; power/per-stream economics should be validated against ASIC encoders/decoders.
- Live and near-real-time streaming in controlled environments (low-delay mode)
- Sectors: live events, enterprise broadcasts, telemedicine, esports.
- What to do: Deploy DCVC-UF (LD, chunk size 1) to meet tight latency budgets while sustaining 60–120 fps at 1080p on commodity GPUs; use in private networks or app-to-app streams where both ends run the DCVC-UF decoder.
- Tools/workflows: Integrate into WebRTC/SRT ingest pipelines on GPU servers; provide software decoders in clients (desktop/mobile apps).
- Assumptions/dependencies: Interoperability with browsers/CE devices is limited until standardized. Savings (≈9.5% vs. VTM-LD) favor speed; choose QP/chunk to meet quality targets.
- Cloud-scale video analytics ingest and archival
- Sectors: smart cities, security/CCTV, retail analytics, sports analytics.
- What to do: Ingest thousands of streams via GPU nodes, compress with DCVC-UF HT modes to reduce storage and bandwidth; decode bursts on demand for retrieval and analysis.
- Tools/workflows: Deploy on GPU clusters with asynchronous chunk-by-chunk pipelines; optionally store intermediate features per chunk to accelerate post-hoc analytics.
- Assumptions/dependencies: Server-side GPU availability; analytics pipelines need either pixel decode or feature adaptors; energy/latency budgets must be evaluated vs. H.265/AV1 pipelines with hardware support.
- Fast-turnaround post-production and collaboration proxies
- Sectors: post-production, newsrooms, sports replay.
- What to do: Generate proxy files quickly (hundreds of FPS at 1080p, substantial at 4K) for editing or review workflows; switch to lossless or high-PSNR settings where required.
- Tools/workflows: FFmpeg-based proxy generation on editorial servers; shared storage backed by DCVC-UF transcodes.
- Assumptions/dependencies: Editors need compatible playback tools; integration into NLEs requires plugin development.
- Cloud gaming and interactive streaming (pilot deployments)
- Sectors: cloud gaming, remote desktop, virtual classrooms.
- What to do: Use DCVC-UF (LD) for server-side encoding to meet frame budget; run software decoders in client apps where possible.
- Tools/workflows: Combine with ABR controllers; integrate with existing streaming stacks as a parallel path to NVENC for A/B trials.
- Assumptions/dependencies: End-to-end control over clients is needed; hardware decoder support is not available, and energy per stream may differ from ASIC encoders.
- Machine learning dataset compression and fast loading
- Sectors: AI/ML infra, academia, autonomous driving, medical imaging (video).
- What to do: Compress large video datasets with DCVC-UF to cut storage while decoding fast on GPU; integrate GPU decoding into training dataloaders to avoid CPU I/O bottlenecks.
- Tools/workflows: PyTorch/TF dataloaders calling DCVC-UF decoders; on-the-fly chunk decoding; persistent caching of chunk features for repeated epochs.
- Assumptions/dependencies: Training/experiments need GPU access and software decoders; ensure quality is adequate for downstream model performance.
- Enterprise archiving and compliance
- Sectors: finance, government, education (lecture capture), healthcare archiving (non-diagnostic).
- What to do: Accelerate backlog transcoding and archival migration using DCVC-UF HT modes to reduce footprint and egress costs.
- Tools/workflows: Scheduled jobs on GPU servers; checksum pipelines; retention policies.
- Assumptions/dependencies: Regulatory constraints may require standards-based codecs or lossless settings for certain assets; verify chain-of-custody and reproducibility.
Long-Term Applications
These require further research, integration, standardization, or hardware/edge maturation.
- Standardization and ecosystem adoption (learned codec as a standard)
- Sectors: consumer devices, browsers, broadcast, OTT.
- What could emerge: A standardized bitstream and conformance tests for learned video codecs; hardware decoder IP in SoCs; browser support via WebCodecs/WebGPU.
- Assumptions/dependencies: Multi-year standardization cycles; power/performance targets on mobile; robust licensing/royalty frameworks; security and DRM integration.
- On-device and edge deployments (mobile cameras, IoT, drones, AR glasses)
- Sectors: mobile, robotics, UAVs, wearables, industrial IoT.
- What could emerge: NPUs/DSPs running quantized/pruned DCVC-UF variants for in-camera compression; hybrid edge–cloud pipelines.
- Assumptions/dependencies: Further model compression, mixed-precision, and operator support on NNAPI/CoreML/Hexagon; energy benchmarking vs. existing ASIC codecs.
- Adaptive chunk sizing and content-aware latency control
- Sectors: streaming, conferencing, surveillance.
- What could emerge: Per-content dynamic chunk sizes trading delay vs. compression; integration with ABR to jointly optimize quality, bitrate, and latency.
- Assumptions/dependencies: Research on chunk adaptivity and control signals; retraining or fine-tuning for stable behavior; complexity of scheduling in multi-tenant pipelines.
- Latent-domain analytics and retrieval
- Sectors: video search, compliance monitoring, privacy-preserving analytics.
- What could emerge: Running detection, tracking, or retrieval on chunk features (F_i) to reduce decode cost and exposure of raw pixels; new “compressed-domain” analytics APIs.
- Assumptions/dependencies: Empirical validation that latent features preserve task-relevant signals; privacy and governance reviews; co-design with downstream models.
- Integration with generative video models and codecs
- Sectors: creative tools, media production, video messaging.
- What could emerge: Unified autoencoders serving both compression and generation tokenization; streaming-friendly latent formats for diffusion/transformer video models.
- Assumptions/dependencies: Training on longer sequences (authors fine-tune to 128; potential up to 1024 with suitable datasets); convergence and stability on diverse content; IPR around joint tokenizers.
- Browser and thin-client delivery via WebGPU/WASM
- Sectors: web apps, education, enterprise portals.
- What could emerge: WASM/WebGPU decoders providing software playback in browsers; progressive loading by chunks for quick seek/play.
- Assumptions/dependencies: Performance of WebGPU across devices; memory constraints; security sandboxing; user consent and feature policy.
- Energy-aware transcoding and sustainability policies
- Sectors: data centers, telcos, hyperscalers.
- What could emerge: Policies leveraging learned codecs to reduce network traffic and storage while maintaining quality; carbon-aware schedulers that place transcodes on efficient GPUs.
- Assumptions/dependencies: Holistic measurements of energy/bit vs. ASIC pipelines; balancing GPU power draw with bitrate savings; carbon accounting standards.
- Teleoperation and immersive XR/telepresence
- Sectors: robotics, healthcare (remote operation), industrial support, XR.
- What could emerge: High-FPS, low-latency neural compression for 3D/VR video feeds with multi-view scaling; hybrid rendering + compression pipelines.
- Assumptions/dependencies: End-to-end latency engineering with sensors and displays; device support for decoders; integration with SLAM/pose streaming where relevant.
- Secure and privacy-preserving media pipelines
- Sectors: regulated industries (healthcare, finance, public sector).
- What could emerge: Architectures where only chunk latents are handled/stored in some stages, minimizing raw pixel exposure; differential privacy or encrypted latents.
- Assumptions/dependencies: Formal privacy guarantees; cryptographic integration and key management; policies and audits aligned with regulations (e.g., HIPAA/GDPR).
In all cases, feasibility and ROI depend on controlled decoder availability, GPU/AI-accelerator access, and workflow integration. For latency-sensitive use, prefer DCVC-UF (LD). For maximum savings or throughput where some delay is acceptable, use DCVC-UF (HT-L/HT-S). The fixed chunk size noted in the paper is a current limitation; adaptive chunking is a promising future enhancement.
Glossary
- Anchor: A reference codec or setting used as the baseline for comparison in metrics like BD-Rate. "where the VTM (Low-Delay, LD) is as the anchor for BD-Rate calculation."
- Arithmetic encoder/decoder: Components of entropy coding that compress/decompress symbols using probabilistic intervals. "DC Block, Q, AE and AD represent depth-wise convolution block, quantization, arithmetic encoder and decoder, respectively."
- Auto-regressive decoding: A sequential entropy decoding strategy where each symbol depends on previously decoded symbols, often slow in practice. "Early NVCs \cite{DCVC, lu2020end} typically employ auto-regressive decoding \cite{minnen2018joint}, which is inherently slow due to its sequential nature."
- BD-Rate: Bjøntegaard Delta Rate; a measure of average bitrate difference between two rate–distortion curves. "Compression ratio is measured by BD-Rate \cite{bd-rate}, where positive values indicate a bitrate increase and negative values indicate savings."
- Bidirectional temporal prediction: Using both past and future frames to predict a target frame for better compression. "by introducing the bidirectional temporal prediction in a GOP (group of pictures)."
- Bit-stream: The serialized coded data produced by the encoder and consumed by the decoder. "This allows bit-stream interactions to be consolidated into a single step, substantially reducing operational cost and improving bit-stream decoding efficiency."
- Bits per pixel (BPP): A rate metric measuring the average number of bits used to represent each pixel. "BPP means bits per pixel."
- Chunk-based coding: A paradigm that encodes/decodes multiple frames together as a unit to improve throughput and temporal modeling. "This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off."
- Conditional coding: A compression approach where current data is coded conditioned on flexible side information (e.g., features) rather than only pixel-domain prediction. "The emerging conditional coding \cite{liu2020conditional, ladune2021conditional,canfvc,vct, DCVC,sheng2021temporal, DCVC-HEM, qi2023motion, DCVC-DC, DCVC-FM, qi2024long} shows larger potential as the temporal context is not limited to pixel-domain prediction but can be any flexible feature."
- Cross-frame interaction modules: Network components that jointly model spatial–temporal relationships across frames within a chunk. "This is enabled by cross-frame interaction modules for joint spatial-temporal modeling"
- Delay-Relaxed (coding): A setting where frames can reference both past and future frames, increasing delay but enabling better compression. "Delay-Relaxed Neural Video Compression"
- Depth-wise convolution: A convolution type applying separate spatial filters per input channel to reduce computation. "DC Block, Q, AE and AD represent depth-wise convolution block, quantization, arithmetic encoder and decoder, respectively."
- Entropy coding: The process of losslessly compressing quantized symbols based on their probability distributions. "we propose a streamlined entropy coding mechanism that consolidates bit-stream interactions into a single step"
- Entropy model: The learned probabilistic model that predicts distribution parameters (e.g., means/scales) for entropy coding. "Our streamlined entropy model allows for better GPU utilization, combined with our chunk-based coding framework, enables DCVC-UF to achieve unprecedented decoding speed."
- Feature propagation: Passing latent features forward across time to exploit temporal redundancy for better compression. "the enabling of the feature propagation mechanism in the latent space, which implicitly captures temporal correlations across multiple frames through joint training."
- Gaussian Splatting: An explicit graphics/video representation using 2D Gaussians for efficient rendering and compression. "Gaussian Splatting \cite{kerbl20233d}"
- GOP (group of pictures): A video coding structure defining a sequence of frames and their temporal prediction relationships. "in a GOP (group of pictures)."
- Hierarchical-B coding: A GOP structure with B-frames arranged hierarchically to enable bidirectional prediction and bitrate savings. "For traditional codec H.266/VTM, the hierarchical-B coding can achieve an average of 33.8\% bitrate saving over the low-delay coding"
- Hyper-prior: An auxiliary latent (prior) that models the distribution of primary latents to improve entropy coding. "takes the prior input (derived from hyper-prior and temporal context )"
- Implicit neural representation (INR): A neural function fitted to data (e.g., a video) that maps coordinates to signal values, enabling compression or rendering. "implicit neural representation (INR) \cite{sitzmann2020implicit}"
- Intra-period: The interval between intra-coded frames that reset prediction chains in a sequence. "Intra-period=--1 for all codecs and settings."
- Latent representation: A compact learned feature encoding of input data used for compression and reconstruction. "encode a chunk of multiple frames into a single compact latent representation"
- Low-delay coding: A configuration restricting references to previously decoded frames only, minimizing delay. "Low-delay coding constrains the coding of the current frame to reference only previously decoded frames in temporal order"
- MACs (Multiply–Accumulate operations): A compute metric counting the number of multiply-accumulate operations, often proxying computational cost. "Average MACs/frame"
- Mixture of Experts (MoE): An architecture that routes inputs to specialized expert modules, here analogous to frame-specific decoders. "This design shares some similarities with Mixture of Experts (MoE) \cite{riquelme2021scaling} architecture"
- Motion compensation: Predicting a frame by warping/aligning reference frames using motion to reduce residual energy. "motion estimation, motion entropy coding, and compensation processes"
- Motion estimation: The process of computing motion (e.g., optical flow or block vectors) between frames for prediction. "motion estimation, motion entropy coding, and compensation processes"
- Motion vector: A coded displacement field indicating how pixels or blocks move between frames for temporal prediction. "each frame relies on explicit motion vector for temporal prediction."
- Patch embedding: Mapping image patches to a lower-resolution or tokenized representation before further processing. "transforming it to 1/8 resolution via patch embedding."
- PSNR: Peak Signal-to-Noise Ratio; a fidelity metric measuring reconstruction quality (in dB). "Video quality is reported using PSNR, with all frames evaluated in YUV420 colorspace."
- Quadtree partition-based entropy coding: A scheme partitioning latents into hierarchical regions to model spatial–channel dependencies during entropy decoding. "The recent quadtree partition-based method in \cite{DCVC-DC} reduces decoding steps to four"
- Quantization: Discretizing continuous latent values to a finite set of levels prior to entropy coding. "This latent is then quantized ()"
- Quantization parameter (QP): A control knob that adjusts quantization strength to trade off rate and distortion. " is the input quantization parameter."
- Rate–distortion–complexity trade-off: The balance between bitrate, reconstruction quality, and computational cost in codec design. "designed to significantly improve the rate-distortion-complexity trade-off."
- Residual coding: Encoding the difference (residual) between a prediction and the true frame. "Many methods \cite{lu2019dvc, hu2020improving,Rippel_2021_ICCV,lu2020end,lin2020m,agustsson2020scale, liu2020neural, liu2023mmvc, ma2024uncertainty} adopt a residual coding inspired by traditional codecs"
- Spatial-channel correlation: Dependency patterns across spatial locations and feature channels leveraged for improved entropy modeling. "to keep the spatial-channel correlation modeling capability."
- Spatial-temporal autoencoder: A neural encoder–decoder that jointly compresses spatial and temporal information into latents for video. "motivated by the spatial-temporal autoencoder \cite{habibian2019video}"
- State of the art (SOTA): The best-known performance level achieved by a method. "sets a new SOTA in performance."
- Temporal chunk context: Propagated information summarizing previous chunk(s) to condition current chunk encoding/decoding. "conditioned on the temporal chunk context "
- YUV420 colorspace: A chroma-subsampled color format commonly used in video coding and evaluation. "Video quality is reported using PSNR, with all frames evaluated in YUV420 colorspace."
Collections
Sign up for free to add this paper to one or more collections.