Learned Multiview Video Coding (LMVC)

• The paper introduces LMVC, an end-to-end learned multiview video coding framework that exploits inter-view priors to reduce inter-view redundancy without explicit disparity estimation.
• It employs a hierarchical B-frame structure and conditions dependent views on decoded independent-view features to ensure random access and backward compatibility.
• Empirical results show significant BD-rate reductions and improved reconstruction quality, making LMVC promising for immersive 3D, VR, and multiview applications.

Searching arXiv for the LMVC paper and closely related codecs mentioned in the provided data. Learned Multiview Video Coding (LMVC) is an end-to-end learned multiview video coding framework for compressing synchronized multiview video under the system constraints of random access and backward compatibility. It is designed for volumetric and immersive 3D applications, where overlapping camera views contain substantial inter-view redundancy but also impose stringent storage and transmission demands. LMVC encodes an independent view with a standard learned single-view codec and encodes dependent views with additional inter-view conditioning, using decoded motion and content features from the independent view to improve compression efficiency without requiring explicit disparity or depth estimation. The framework is reported as the first end-to-end learned multiview video coding framework that simultaneously achieves random access, backward compatibility of the independent view, and strong compression gains through inter-view motion and content correlation modeling (Sheng et al., 4 Sep 2025).

1. Problem formulation and system constraints

LMVC addresses multiview video coding (MVC), whose objective is to reduce redundancy across both time and cameras. In the setting targeted by LMVC, many synchronized cameras capture overlapping views of a dynamic scene; this is characteristic of free-viewpoint sports, VR, and cinematic capture. Independent per-view coding is inefficient because it ignores strong inter-view redundancy, while classical multiview pipelines rely heavily on handcrafted tools. LMVC reframes MVC with learned modules and end-to-end rate–distortion (RD) optimization, motivated by the scale of volumetric video and by the observation that large camera counts and high resolutions make handcrafted tools less effective than learned compression (Sheng et al., 4 Sep 2025).

The framework is organized around two view classes. The independent view, denoted $V_0$, is coded standalone using DCVC-B. Its bitstream is self-sufficient and provides decoded inter-view features for other views. A dependent view, denoted $V_i$ for $i>0$, is coded with additional inter-view conditioning and is decoded given the independent view’s decoded features. This separation implements a learned analogue of the base/dependent view structure used in multiview standards.

Two system-level requirements are explicit. Random access means that a decoder can start at designated access points without decoding long temporal histories. Backward compatibility means that the independent-view bitstream can be decoded by an ordinary single-view decoder and does not depend on side information from dependent views. LMVC preserves these requirements by retaining a hierarchical B-frame structure with an intra-period and by ensuring that dependent views never constrain decoding of the independent view.

Temporal organization follows a hierarchical B-frame GOP with an intra-period of 32 and 6 temporal layers; simplified diagrams use intra-period 8 and 4 layers. Across cameras, a typical 3-view coding order is center–left–right, and a 2-view order is right–left, consistent with MV-HEVC common test conditions. This organization matters because LMVC is not merely a view-conditioned learned codec; it is a codec explicitly shaped by multiview deployment constraints.

2. End-to-end pipeline and bitstream organization

The LMVC pipeline operates per view. For motion estimation, SpyNet estimates pixel-wise bi-directional flows $v_{t\rightarrow f}^{V_i}$ and $v_{t\rightarrow b}^{V_i}$ between the current frame $x_t^{V_i}$ and its forward and backward temporal references. DCVC-B supplies bi-directional motion difference features for $V_0$ and temporal references in $V_i$. Motion compression then encodes residual motion differences $r_{t\rightarrow f}^{V_i}$ and $r_{t\rightarrow b}^{V_i}$ with a motion encoder–decoder conditioned on temporal motion features from the dependent view’s references and on decoded inter-view motion features from the independent view. Content feature extraction produces low-resolution content features $V_i$0 and content latents $V_i$1, with bidirectional temporal context mining from reference features and flows, while an inter-view context is predicted from $V_i$2’s decoded content features without explicit disparity. Motion and content latents are quantized and entropy coded with hyperpriors, temporal priors, and inter-view priors, and decoding reconstructs motion vectors, content features, and frames within the hierarchical B structure (Sheng et al., 4 Sep 2025).

The bitstream design separates independent-view and dependent-view signaling. The $V_i$3 bitstream is a pure DCVC-B bitstream containing I/B-coded content latents $V_i$4, motion latents $V_i$5, and corresponding hyper-latents $V_i$6, with no multiview dependencies. Each dependent-view bitstream carries its own motion latents $V_i$7, content latents $V_i$8, and hyper-latents $V_i$9 and $i>0$0. Inter-view conditioning uses decoded features from $i>0$1 that are already available at the decoder, so no extra cross-view side data needs to be signaled in the dependent-view bitstream.

This yields the following functional decomposition:

Component	Conditioning source	Role
Independent view $i>0$2	None beyond DCVC-B temporal coding	Self-sufficient base bitstream
Dependent-view motion coding	Temporal motion features and decoded $i>0$3 motion features	Reduces motion rate
Dependent-view content coding	Temporal contexts and decoded $i>0$4 content features	Reduces content rate

The random-access property follows from the hierarchical B-frame GOP with intra-period 32, which establishes access points. Independent-view access points can be decoded independently. Dependent views reference temporal neighbors within the same view and use same-timestamp inter-view features from $i>0$5, but the decoding of $i>0$6 is never impacted by dependent-view data. Backward compatibility follows directly from the fact that the $i>0$7 bitstream remains a pure DCVC-B stream that any DCVC-B decoder can decode.

3. Inter-view motion and content modeling

The principal technical contribution of LMVC is the explicit use of decoded independent-view information as priors for dependent-view motion and content coding. On the motion side, the framework introduces feature-based inter-view motion vector prediction (IVMVP). The premise is that motion fields across neighboring views are correlated. Instead of coding dependent-view motion only from temporal predictors, LMVC conditions motion encoding on decoded bi-directional motion difference features from $i>0$8. If $i>0$9 and $v_{t\rightarrow f}^{V_i}$0 denote the residual motion differences, the motion encoder receives temporal motion features from $v_{t\rightarrow f}^{V_i}$1’s references together with inter-view motion difference features from $v_{t\rightarrow f}^{V_i}$2 at time $v_{t\rightarrow f}^{V_i}$3.

The inter-view motion feature fusion varies with the reference-picture types:

$v_{t\rightarrow f}^{V_i}$4

when both references are I-frames;

$v_{t\rightarrow f}^{V_i}$5

for forward I and backward B;

$v_{t\rightarrow f}^{V_i}$6

for forward B and backward I; and

$v_{t\rightarrow f}^{V_i}$7

when both references are B-frames. Here $v_{t\rightarrow f}^{V_i}$8 denotes channel-wise concatenation, $v_{t\rightarrow f}^{V_i}$9 are depth-block fusion adaptors, and $v_{t\rightarrow b}^{V_i}$0 denotes decoded motion difference features.

LMVC complements IVMVP with an inter-view motion entropy model (IVMEM). It extracts a feature from $v_{t\rightarrow b}^{V_i}$1’s decoded bi-directional motion vectors,

$v_{t\rightarrow b}^{V_i}$2

then forms an inter-view motion prior

$v_{t\rightarrow b}^{V_i}$3

This inter-view prior is aggregated with a temporal motion prior, derived from $v_{t\rightarrow b}^{V_i}$4 and $v_{t\rightarrow b}^{V_i}$5, and with a motion hyperprior from $v_{t\rightarrow b}^{V_i}$6. A quadtree partition-based spatial context model then estimates the Gaussian mean and scale parameters for the motion latent $v_{t\rightarrow b}^{V_i}$7:

$v_{t\rightarrow b}^{V_i}$8

On the content side, LMVC introduces disparity-free inter-view context prediction (IVCP). Its stated goal is to exploit inter-view content correlation without estimating explicit disparity or depth, thereby avoiding the additional complexity that would be imposed on top of already expensive bi-directional flow estimation in B-frame coding. The module operates on low-resolution decoded content features $v_{t\rightarrow b}^{V_i}$9 of spatial size $x_t^{V_i}$0, motivated by the observation that downsampling alleviates misalignment. The inter-view context is

$x_t^{V_i}$1

where $x_t^{V_i}$2 modifies the temporal context mining feature extractor by removing downsampling to preserve resolution. This context is then combined with multiscale temporal contexts $x_t^{V_i}$3, $x_t^{V_i}$4, $x_t^{V_i}$5, $x_t^{V_i}$6, $x_t^{V_i}$7, and $x_t^{V_i}$8 in the contextual encoder–decoder.

The corresponding inter-view contextual entropy model (IVCEM) derives a content-prior feature

$x_t^{V_i}$9

forms the inter-view content prior

$V_0$0

and combines it with temporal context priors and a context hyperprior $V_0$1 to predict $V_0$2 for the content latent $V_0$3:

$V_0$4

A common assumption in multiview coding is that strong inter-view exploitation requires explicit geometry or disparity; LMVC is positioned against that assumption by showing a disparity-free pathway based on decoded low-resolution features and latent priors (Sheng et al., 4 Sep 2025).

4. Rate–distortion objective, priors, and training protocol

LMVC trains the dependent-view codecs under a hierarchical B-frame quality schedule using the RD objective

$V_0$5

Here $V_0$6 is the distortion between the original frame $V_0$7 and its reconstruction $V_0$8, measured by MSE:

$V_0$9

The rate terms $V_i$0 and $V_i$1 are the expected code lengths of the motion and content latent streams, including hyper-latents, computed by the entropy model via $V_i$2 and summed over spatial positions and channels. Variable-rate is realized with learnable quantization steps. The temporal-layer weight $V_i$3 enforces hierarchical quality across B layers, and the Lagrange multiplier $V_i$4 controls the RD trade-off; the values used in experiments are $V_i$5 (Sheng et al., 4 Sep 2025).

The entropy architecture comprises motion latents $V_i$6 with motion hyper-latents $V_i$7 and content latents $V_i$8 with contextual hyper-latents $V_i$9. Hyperpriors compress the hyper-latents for both motion and content. Temporal priors are view-specific: for motion they are supplied by the bi-directional temporal motion vectors $r_{t\rightarrow f}^{V_i}$0 and $r_{t\rightarrow f}^{V_i}$1 between the dependent view’s forward and backward reference frames; for content they are supplied by bi-directional multiscale temporal contexts mined through motion-compensated features from those references. Inter-view priors are built from decoded $r_{t\rightarrow f}^{V_i}$2 motion vectors, motion latents, inter-view context, and content latents. The quadtree partition-based spatial context model combines these priors to predict $r_{t\rightarrow f}^{V_i}$3 and $r_{t\rightarrow f}^{V_i}$4 for each latent element, replacing masked-convolution autoregression with a spatially adaptive partitioning scheme.

Because authentic multiview training video is scarce, the training pipeline synthesizes multiview pairs and triples from single-view clips using affine and homography transformations with randomized displacements and perspective parameters. The stated purpose is to generate geometrically consistent multiview sequences that mimic cross-view correlations. Training uses 7-frame Vimeo-90k and 33-frame DCVC-B video clips processed through this synthesis pipeline. Testing uses standard MV-HEVC multiview sequences in 2-view and 3-view configurations, including Poznan_Hall2, Poznan_Street, Undo_Dancer, GT_Fly, Shark, Balloons, Newspaper1, and Kendo. The coding order is center–left–right for 3-view and right–left for 2-view. The independent-view DCVC-B is frozen during training; the optimizer is AdamW with batch size 8; hierarchical temporal weights are $r_{t\rightarrow f}^{V_i}$5 for B layers; the I-frame codec is shared with DCVC-B; the intra-period is 32; 97 frames are coded per sequence; and complexity measurements are reported on an NVIDIA RTX 3090.

5. Empirical performance and ablation evidence

The reported quantitative results use HTM-16.3 as the MV-HEVC anchor under baseCfg_3view and baseCfg_2view with intra-period 32 and 97 coded frames. In the 3-view configuration, LMVC achieves an average BD-rate reduction of $r_{t\rightarrow f}^{V_i}$6 relative to HTM, with sequence-level highlights of $r_{t\rightarrow f}^{V_i}$7 on Poznan_Hall2, $r_{t\rightarrow f}^{V_i}$8 on Balloons, and $r_{t\rightarrow f}^{V_i}$9 on Kendo. The paper also reports that LMVC improves over DCVC-B by more than $r_{t\rightarrow b}^{V_i}$0 BD-rate on average. In the 2-view configuration, LMVC achieves an average BD-rate reduction of $r_{t\rightarrow b}^{V_i}$1, compared with DCVC-B’s $r_{t\rightarrow b}^{V_i}$2 average, and the RD curves are described as dominating across most rate points for natural content (Sheng et al., 4 Sep 2025).

Configuration	Result
3-view vs HTM-16.3	Average BD-rate reduction: $r_{t\rightarrow b}^{V_i}$3
2-view vs HTM-16.3	Average BD-rate reduction: $r_{t\rightarrow b}^{V_i}$4
3-view highlights	Poznan_Hall2: $r_{t\rightarrow b}^{V_i}$5; Balloons: $r_{t\rightarrow b}^{V_i}$6; Kendo: $r_{t\rightarrow b}^{V_i}$7

The ablation study proceeds from baseline $r_{t\rightarrow b}^{V_i}$8 to full model $r_{t\rightarrow b}^{V_i}$9 and isolates the effect of the inter-view modules. The sequence of gains is reported as follows: IVMP alone yields a $V_i$00 BD-rate gain; IVMP + IVMEM yields $V_i$01; adding IVCP yields $V_i$02; and adding IVCEM yields $V_i$03. The reported interpretation is additive: motion-side inter-view priors reduce motion bitrate, and content-side inter-view context reduces content bitrate.

Bitrate component analysis further localizes the savings. For motion coding, LMVC requires $V_i$04–$V_i$05 of DCVC-B’s motion bitrate on Balloons and $V_i$06–$V_i$07 on Kendo across rate points. For content coding, LMVC requires $V_i$08–$V_i$09 of DCVC-B’s content bitrate on Balloons and $V_i$10–$V_i$11 on Kendo. These values indicate that the motion-side gains are proportionally larger, while content-side savings remain substantial.

The complexity overhead is modest relative to the baseline codec. For a $V_i$12 dependent-view frame, encoder time is $V_i$13s for LMVC versus $V_i$14s for DCVC-B, and decoder time is $V_i$15s versus $V_i$16s. The reported MACs per pixel are $V_i$17K for LMVC and $V_i$18K for DCVC-B; parameter counts are $V_i$19M and $V_i$20M, respectively. Qualitatively, reconstructed frames are described as exhibiting crisper textures and fewer artifacts than HTM and DCVC-B, consistent with the RD results.

6. Positioning, limitations, and prospective developments

LMVC is positioned between single-view learned codecs and classical multiview standards. Relative to single-view learned codecs such as the DVC/DCVC family, it augments temporal learned compression with inter-view conditioning. The additions explicitly identified are feature-based inter-view motion prediction conditioned on $V_i$21’s motion features and vectors, disparity-free inter-view content context predicted from $V_i$22’s decoded low-resolution content features, and inter-view priors integrated into entropy models for both motion and content. Relative to MV-HEVC, which uses block-based disparity and motion tools and may use explicit depth layers in 3D-HEVC, LMVC replaces hand-engineered tools with neural modules optimized end to end. The framework is specifically described as achieving large bitrate savings without explicit depth or disparity, which simplifies the pipeline and reduces estimation overhead (Sheng et al., 4 Sep 2025).

The advantages of the disparity-free design are framed in computational as well as coding terms. By not estimating cross-view disparity or depth, LMVC avoids adding another expensive estimation stage on top of bi-directional temporal flow. By operating on low-resolution decoded features, it uses the claim that downsampling alleviates misalignment to capture cross-view correlation with reduced compute. This suggests a design preference for feature-domain cross-view alignment over explicit geometric estimation when the system already carries substantial temporal modeling cost.

The limitations are also explicit. Synthetic or animation-style sequences such as Shark and Undo_Dancer exhibit domain shift relative to the natural video training data, and both LMVC and DCVC-B can be worse than HTM in such cases. A concrete example reported is that LMVC has $V_i$23 BD-rate on Shark, while DCVC-B has $V_i$24. The paper further notes that gains may diminish when inter-view correlation is weak, camera baselines are large, occlusions are strong, or synchronization is imperfect. These caveats are important because they delimit the regime in which inter-view conditioning is most effective.

The proposed future directions are geometric and scalable. They include adding an explicit depth map coding layer to further exploit geometry, scaling to more views through dynamic inter-view graph conditioning, and integrating the codec with volumetric representations such as point clouds and neural radiance fields for end-to-end immersive media pipelines. A plausible implication is that LMVC is intended not as an isolated codec design, but as a component in broader learned representations for immersive media, where multiview video serves as a bridge between conventional compressed video and volumetric scene representations.