BRHVC: Bi-directional Harmonized Video Compression

• BRHVC is a neural B-frame video compression method that explicitly targets the unbalanced contributions of forward and backward references in hierarchical coding.
• It introduces Bi-directional Motion Converge (BMC) to compress multi-scale optical flows and Bi-directional Contextual Fusion (BCF) to adaptively weight reference contexts based on motion-compensation quality.
• BRHVC achieves state-of-the-art performance on HEVC datasets, delivering up to 44.7% bitrate savings and demonstrating robust improvements through ablation studies.

Bi-directional Reference Harmonization Video Compression (BRHVC) is a neural B-frame video compression method for hierarchical random-access coding that is designed to exploit a forward reference and a backward reference more effectively than earlier neural video compression pipelines. Its defining premise is that, in hierarchical B-frame coding, the two references are often not equally informative, especially at large frame spans; BRHVC addresses this with Bi-directional Motion Converge (BMC) for motion compression and Bi-directional Contextual Fusion (BCF) for adaptive reference-context weighting under the guidance of motion-compensation accuracy (Liu et al., 12 Nov 2025).

1. Problem setting and core motivation

BRHVC is formulated for neural B-frame video compression (NBVC) in a hierarchical random-access (RA) structure. In this setting, the current frame is coded with access to two decoded references, one in each temporal direction. The paper emphasizes that this differs materially from ordinary P-frame compression, where only a previous frame is referenced and the temporal dependency is simpler (Liu et al., 12 Nov 2025).

The central difficulty is unbalanced reference contribution (URC). In large-span hierarchical levels, one reference frame may be much more useful than the other, and in some regions both references may be unreliable. The method is motivated by the observation that existing NBVC systems often fuse the two reference contexts too naively—typically by concatenation—thereby implicitly assuming equal importance. BRHVC is explicitly designed to avoid that assumption by modeling reference usefulness spatially and temporally (Liu et al., 12 Nov 2025).

The hierarchical source of URC is stated directly. For IP$=32$, the frame span at a given hierarchical level is

$b-f = 2^{6-\text{level}}$

where $b$ and $f$ refer to backward and forward reference positions. The paper reports a BD-rate experiment on a single frame across spans $\{32,16,8,4,2\}$, with the gap between the better and worse reference reaching 20.6% at span 32 and 11.1% at span 16. These values are used to support the claim that a B-frame codec should not treat both references as equally useful everywhere (Liu et al., 12 Nov 2025).

A concrete example given in the paper is a racehorse scene in which a number plate is visible in only one reference because the other reference is occluded. In that scenario, equal-weight fusion wastes bits and harms reconstruction. BRHVC therefore frames “harmonization” not as symmetric blending, but as selective use of bidirectional evidence, with the possibility of reducing reliance on both references when neither is reliable (Liu et al., 12 Nov 2025).

2. Position within learned hierarchical bi-directional compression

BRHVC belongs to a line of learned hierarchical bi-directional codecs that use both past and future references for B-frame coding, but it makes “reference harmonization” the explicit organizing principle. Earlier work on flexible-rate learned hierarchical bi-directional video compression already combined a keyframe-based GoP structure with B-frames coded from past and future decoded references, along with motion prediction, motion refinement, mask-based fusion, and gain-based rate control. In that framework, for a current frame $x_c$, the codec uses $\hat{x}_p$ and $\hat{x}_f$, warps both toward the current frame, and fuses them as a motion-compensated estimate before residual coding (Cetin et al., 2022).

That earlier family is directly relevant because it already embodies the main bidirectional coding pattern: dual-reference usage, learned fusion, and hierarchical B-picture compression. Its motion-compensated fusion is written as

$$\hat{x}^{comp}= \dfrac{m_1}{m_1+m_2}\odot W_b(\hat{x}_p, \hat{mv}_{c\rightarrow p}) + \dfrac{m_2}{m_1+m_2}\odot W_b(\hat{x}_f, \hat{mv}_{c\rightarrow f}),$$

which makes the two-reference reconciliation explicit in the pixel domain (Cetin et al., 2022).

A later extension moved the harmonization process into feature space through multi-scale deformable alignment and multi-scale conditional coding. There, the current frame $x_t$ is compressed using deformably aligned feature maps from $b-f = 2^{6-\text{level}}$0 and $b-f = 2^{6-\text{level}}$1, with the aligned bidirectional prediction acting as a condition for coding the current frame. The aligned features are computed as

$b-f = 2^{6-\text{level}}$2

and the current-frame representation is coded conditionally on those aligned features (Yılmaz et al., 2023).

Within this progression, BRHVC can be understood as a method that retains hierarchical bidirectional coding but shifts the focal problem from general bidirectional prediction to the specific asymmetry of bidirectional reference utility. This suggests a narrower and more explicit formulation of harmonization than in the earlier flexible-rate and deformable-alignment codecs: BRHVC is not only using two references, but is designed around the claim that their contributions are systematically unequal under hierarchical RA coding (Liu et al., 12 Nov 2025).

3. BRHVC pipeline and Bi-directional Motion Converge

Given current frame $b-f = 2^{6-\text{level}}$3, BRHVC uses decoded forward reference $b-f = 2^{6-\text{level}}$4, decoded backward reference $b-f = 2^{6-\text{level}}$5, and their features $b-f = 2^{6-\text{level}}$6. The paper summarizes the pipeline in five stages: motion estimation, motion compression, motion compensation, reference context fusion, and transform and entropy coding. Motion estimation computes multi-scale optical flows from both directions using SpyNet; motion compression uses BMC; motion compensation warps reference features using reconstructed flows; reference contexts are then fused with BCF; and the resulting latent variables are compressed with a quadtree partition-based entropy model (Liu et al., 12 Nov 2025).

BMC is the motion-coding component. Its motivation is that standard motion coding methods such as BMRC compress motion residuals between flows, but this becomes less suitable when the codec uses multi-scale flows and must represent large-span motion. BRHVC instead converges multiple optical flows into a single latent variable, fuses the latent variables from both directions, and reconstructs the flows in the decoder (Liu et al., 12 Nov 2025).

The flow generation stage uses SpyNet to produce three scales of optical flow in each direction:

$b-f = 2^{6-\text{level}}$7

The references and current frame are downsampled twice before optical-flow estimation so as to enlarge the receptive field. The paper states that this is important for large frame spans, where a single flow scale may not carry enough motion information (Liu et al., 12 Nov 2025).

BMC also uses motion-related priors from the decoded buffer: $b-f = 2^{6-\text{level}}$8, which are the motion features of the two references, and $b-f = 2^{6-\text{level}}$9, which is a feature-domain representation of inter-reference motion $b$0 and $b$1. The paper states that $b$2 enriches the prior compared to BMRC. In addition, a Motion Feature Adapter selects different sub-networks depending on frame type, B-frame or I-frame, so that prior motion features are adapted appropriately (Liu et al., 12 Nov 2025).

The claimed significance of BMC is twofold. First, it yields more efficient motions for large frame spans by using multi-scale explicit optical flows and stronger priors. Second, it improves downstream motion compensation, which is then used by BCF to decide how much trust to place in each reference context (Liu et al., 12 Nov 2025).

4. Bi-directional Contextual Fusion and explicit harmonization

BCF is the module that makes BRHVC a reference-harmonization codec in the strict sense. Its stated purpose is to explicitly predict weights for the two reference contexts and combine them adaptively, guided by motion-compensation quality. This is contrasted with prior methods that fuse reference contexts by concatenation, an operation the paper treats as an implicit equal-importance assumption (Liu et al., 12 Nov 2025).

At the encoder, BCF-E uses the current frame $b$3, the warped references $b$4 and $b$5, and reference contexts $b$6 at scales $b$7. The warped references are

$b$8

The encoder-side contextual fusion is then described as

$b$9

The fused contexts themselves are computed explicitly:

$f$0

Here $f$1 and $f$2 are learned forward and backward reference weights, $f$3 is a learned bias or supplementary term for the current frame, $f$4 denotes element-wise multiplication, and $f$5 is a depth block at scale $f$6 (Liu et al., 12 Nov 2025).

The bias term is central to the paper’s definition of harmonization. It allows the model to rely on the current frame’s own features when neither reference is reliable, when both references are weak, or when the target region is not well supported by either reference. Accordingly, BRHVC does not reduce harmonization to choosing the better of two references; it also models cases in which both references should be downweighted (Liu et al., 12 Nov 2025).

At the decoder, $f$7 is unavailable, so the method uses $f$8 together with pixel-feature alignment (PFA) to guide decoder-side fusion:

$f$9

The paper also states that BCF estimates reference weights from pixel-domain similarity between the current frame and the warped forward and backward references. Its visualizations show that, for large spans, the more useful reference receives larger average weight, the bias term becomes larger when compensation is harder, and for small spans the two references become more similar in importance while the bias shrinks (Liu et al., 12 Nov 2025).

5. Optimization, datasets, and reported compression performance

BRHVC is built on the backbone ideas of DCVC-B and DCVC-DC, while replacing single-direction contexts with bi-directional reference contexts. Its encoder/decoder architecture is described as largely like DCVC-DC, with the principal difference being the use of bidirectional contexts (Liu et al., 12 Nov 2025).

The reported training setup is specific. BRHVC is trained on Vimeo-90k with 7-frame sequences, fine-tuned on original Vimeo videos with 17-frame sequences, and tested on HEVC Classes B–E, UVG, and MCL-JCV. The main testing configuration uses IP = 32. The optimizer is AdamW, the batch size is 8, the random crop size is 256×256, random sequence reversal is applied with probability 0.5, and the training strategy is a multi-stage training strategy following DCVC-B (Liu et al., 12 Nov 2025).

The main reported compression results are on HEVC datasets. BRHVC achieves 27.3% average bitrate saving over VTM-LDB, 32.0% average bitrate saving on HEVC datasets, and surpasses VTM-RA, which gets 30.9% average saving on HEVC datasets. The reported gain is especially strong on HEVC Class D, where BRHVC saves 44.7%. The paper also states that BRHVC outperforms the previous state-of-the-art neural methods DCVC-DC, DCVC-FM, and DCVC-B (Liu et al., 12 Nov 2025).

Under MS-SSIM optimization, the method also surpasses DCVC-B, with average -44.0% versus DCVC-B’s -42.4% on HEVC B–E. The paper further reports that BRHVC’s advantage over DCVC-B increases as IP increases: IP8: -15.3% w/o I-frame, IP16: -17.8%, and IP32: -21.6%. This is presented as direct support for the claim that URC becomes more severe as frame span grows and that BRHVC is specifically effective in that regime (Liu et al., 12 Nov 2025).

The paper also notes an important limitation: BRHVC is less strong on high frame-rate datasets such as UVG and MCL-JCV, and it explicitly states that further study is needed there. That caveat constrains the generality of its strongest performance claims, which are centered on the HEVC datasets under random-access conditions (Liu et al., 12 Nov 2025).

6. Ablation evidence, interpretive scope, and related misconceptions

The ablation results isolate the contributions of the two core modules. Starting from a Baseline (BL) that is described as a reference implementation close to DCVC-B, the paper reports that BL + BMC improves by 6.4%, BL + BCF improves by 6.6%, and BL + BCF + BMC improves by 12.3%. These figures are used to argue that BCF directly addresses URC, BMC strengthens motion estimation, and the two are complementary when combined (Liu et al., 12 Nov 2025).

A separate multi-scale flow ablation is reported for BMC. Replacing the largest-scale flows causes a 6.7% BD-rate loss, replacing the intermediate-scale flows causes 4.2%, and replacing the smallest-scale flows causes 3.0%. The paper interprets this as evidence that all scales contribute and that larger-scale flows are especially important for difficult large-span motion. Visualizations are said to show that BMC helps avoid disordered estimated motion in background regions when the span is large (Liu et al., 12 Nov 2025).

A common misconception is to treat BRHVC as merely another bidirectional concatenation model. The paper argues against that view directly. BCF does not simply concatenate forward and backward contexts; it explicitly predicts $\{32,16,8,4,2\}$0 and $\{32,16,8,4,2\}$1, and it supplements them with $\{32,16,8,4,2\}$2. This means that harmonization in BRHVC is not equivalent to equal-weight averaging, nor is it limited to binary selection of one reference over the other. The current-frame supplementary term is part of the method’s formal mechanism (Liu et al., 12 Nov 2025).

Another misconception is to treat BRHVC as isolated from earlier learned hierarchical B-frame codecs. The related literature shows a clear progression from pixel-domain bidirectional warping and mask-based fusion with flexible-rate gain control to feature-level deformable alignment and conditional coding, and then to explicit motion-guided reference harmonization under URC. This suggests that BRHVC is best read as a specialized development within learned hierarchical bi-directional video compression rather than as a departure from that line of work (Cetin et al., 2022, Yılmaz et al., 2023).

Taken together, the evidence supports a specific characterization: BRHVC is a neural B-frame codec whose novelty lies in making the unequal usefulness of two references the primary modeling target. BMC improves the motion representation and motion compensation; BCF uses that improved compensation to weight reference contexts adaptively; and the resulting system achieves state-of-the-art compression performance on the reported HEVC random-access benchmarks, including surpassing VTM-RA on HEVC datasets (Liu et al., 12 Nov 2025).