---
title: 'BRHVC: Bi-directional Harmonized Video Compression'
url: https://www.emergentmind.com/topics/bi-directional-reference-harmonization-video-compression-brhvc
type: topic
---

# BRHVC: Bi-directional Harmonized Video Compression

Bi-directional Reference Harmonization Video Compression (BRHVC) is a neural B-frame video compression method for hierarchical random-access coding that is designed to exploit a forward reference and a backward reference more effectively than earlier neural video compression pipelines. Its defining premise is that, in hierarchical B-frame coding, the two references are often not equally informative, especially at large frame spans; BRHVC addresses this with Bi-directional Motion Converge (BMC) for motion compression and Bi-directional Contextual Fusion (BCF) for adaptive reference-context weighting under the guidance of motion-compensation accuracy [2511.08938].

## 1. Problem setting and core motivation

BRHVC is formulated for neural B-frame video compression (NBVC) in a hierarchical random-access (RA) structure. In this setting, the current frame is coded with access to two decoded references, one in each temporal direction. The paper emphasizes that this differs materially from ordinary P-frame compression, where only a previous frame is referenced and the temporal dependency is simpler [2511.08938].

The central difficulty is **unbalanced reference contribution (URC)**. In large-span hierarchical levels, one reference frame may be much more useful than the other, and in some regions both references may be unreliable. The method is motivated by the observation that existing NBVC systems often fuse the two reference contexts too naively—typically by concatenation—thereby implicitly assuming equal importance. BRHVC is explicitly designed to avoid that assumption by modeling reference usefulness spatially and temporally [2511.08938].

The hierarchical source of URC is stated directly. For IP\(=32\), the frame span at a given hierarchical level is

$$
b-f = 2^{6-\text{level}}
$$

where \(b\) and \(f\) refer to backward and forward reference positions. The paper reports a BD-rate experiment on a single frame across spans \(\{32,16,8,4,2\}\), with the gap between the better and worse reference reaching **20.6%** at span 32 and **11.1%** at span 16. These values are used to support the claim that a B-frame codec should not treat both references as equally useful everywhere [2511.08938].

A concrete example given in the paper is a racehorse scene in which a number plate is visible in only one reference because the other reference is occluded. In that scenario, equal-weight fusion wastes bits and harms reconstruction. BRHVC therefore frames “harmonization” not as symmetric blending, but as selective use of bidirectional evidence, with the possibility of reducing reliance on both references when neither is reliable [2511.08938].

## 2. Position within learned hierarchical bi-directional compression

BRHVC belongs to a line of learned hierarchical bi-directional codecs that use both past and future references for B-frame coding, but it makes “reference harmonization” the explicit organizing principle. Earlier work on flexible-rate learned hierarchical bi-directional video compression already combined a keyframe-based GoP structure with B-frames coded from past and future decoded references, along with motion prediction, motion refinement, mask-based fusion, and gain-based rate control. In that framework, for a current frame \(x_c\), the codec uses \(\hat{x}_p\) and \(\hat{x}_f\), warps both toward the current frame, and fuses them as a motion-compensated estimate before residual coding [2206.13613].

That earlier family is directly relevant because it already embodies the main bidirectional coding pattern: dual-reference usage, learned fusion, and hierarchical B-picture compression. Its motion-compensated fusion is written as

$$
\hat{x}^{comp}= \dfrac{m_1}{m_1+m_2}\odot W_b(\hat{x}_p, \hat{mv}_{c\rightarrow p}) + \dfrac{m_2}{m_1+m_2}\odot W_b(\hat{x}_f, \hat{mv}_{c\rightarrow f}),
$$

which makes the two-reference reconciliation explicit in the pixel domain [2206.13613].

A later extension moved the harmonization process into feature space through **multi-scale deformable alignment** and **multi-scale conditional coding**. There, the current frame \(x_t\) is compressed using deformably aligned feature maps from \(x_{t-i}\) and \(x_{t+i}\), with the aligned bidirectional prediction acting as a condition for coding the current frame. The aligned features are computed as

$$
F^{comp-1,2,3}_{t} = Dcn^{1,2,3}\!\left(c(\hat{F}^{1,2,3}_{t-i}, \hat{F}^{1,2,3}_{t+i}), o^{1,2,3}_{t}, m^{1,2,3}_{t}\right),
$$

and the current-frame representation is coded conditionally on those aligned features [2306.16544].

Within this progression, BRHVC can be understood as a method that retains hierarchical bidirectional coding but shifts the focal problem from general bidirectional prediction to the specific asymmetry of bidirectional reference utility. This suggests a narrower and more explicit formulation of harmonization than in the earlier flexible-rate and deformable-alignment codecs: BRHVC is not only using two references, but is designed around the claim that their contributions are systematically unequal under hierarchical RA coding [2511.08938].

## 3. BRHVC pipeline and Bi-directional Motion Converge

Given current frame \(x_t\), BRHVC uses decoded forward reference \(\hat{x}_f\), decoded backward reference \(\hat{x}_b\), and their features \(F_f, F_b\). The paper summarizes the pipeline in five stages: **motion estimation**, **motion compression**, **motion compensation**, **reference context fusion**, and **transform and entropy coding**. Motion estimation computes multi-scale optical flows from both directions using SpyNet; motion compression uses BMC; motion compensation warps reference features using reconstructed flows; reference contexts are then fused with BCF; and the resulting latent variables are compressed with a quadtree partition-based entropy model [2511.08938].

BMC is the motion-coding component. Its motivation is that standard motion coding methods such as BMRC compress motion residuals between flows, but this becomes less suitable when the codec uses **multi-scale flows** and must represent large-span motion. BRHVC instead converges multiple optical flows into a single latent variable, fuses the latent variables from both directions, and reconstructs the flows in the decoder [2511.08938].

The flow generation stage uses SpyNet to produce three scales of optical flow in each direction:

$$
\{v^1_{t,f}, v^2_{t,f}, v^3_{t,f}\}, \quad \{v^1_{t,b}, v^2_{t,b}, v^3_{t,b}\}.
$$

The references and current frame are downsampled twice before optical-flow estimation so as to enlarge the receptive field. The paper states that this is important for large frame spans, where a single flow scale may not carry enough motion information [2511.08938].

BMC also uses motion-related priors from the decoded buffer: \(\{F^v_f, F^v_b\}\), which are the motion features of the two references, and \(F^{ref}_{f,b}\), which is a feature-domain representation of inter-reference motion \(v_{f,b}\) and \(v_{b,f}\). The paper states that \(F^{ref}_{f,b}\) enriches the prior compared to BMRC. In addition, a **Motion Feature Adapter** selects different sub-networks depending on frame type, B-frame or I-frame, so that prior motion features are adapted appropriately [2511.08938].

The claimed significance of BMC is twofold. First, it yields more efficient motions for large frame spans by using multi-scale explicit optical flows and stronger priors. Second, it improves downstream motion compensation, which is then used by BCF to decide how much trust to place in each reference context [2511.08938].

## 4. Bi-directional Contextual Fusion and explicit harmonization

BCF is the module that makes BRHVC a reference-harmonization codec in the strict sense. Its stated purpose is to **explicitly predict weights for the two reference contexts** and combine them adaptively, guided by motion-compensation quality. This is contrasted with prior methods that fuse reference contexts by concatenation, an operation the paper treats as an implicit equal-importance assumption [2511.08938].

At the encoder, BCF-E uses the current frame \(x_t\), the warped references \(x^{warp}_{t,f}\) and \(x^{warp}_{t,b}\), and reference contexts \(\{C^i_f, C^i_b\}\) at scales \(i=1,2,3\). The warped references are

$$
x^{warp}_{t,f}=warp(\hat{x}_{f},\hat{v}^1_{t,f}), \quad x^{warp}_{t,b}=warp(\hat{x}_{b},\hat{v}^1_{t,b}).
$$

The encoder-side contextual fusion is then described as

$$
C^1_t, C^2_t, C^3_t = BCF_E(\{C^1_f,C^1_b\}, \{C^2_f,C^2_b\}, \{C^3_f,C^3_b\} \mid \{x_t, x^{warp}_{t,f}, x^{warp}_{t,b}\}).
$$

The fused contexts themselves are computed explicitly:

$$
\begin{aligned}
C^1_t &= DB_1(s^1_f \otimes C^1_f + s^1_b \otimes C^1_b + b^1_t), \\
C^2_t &= DB_2(s^2_f \otimes C^2_f + s^2_b \otimes C^2_b + b^2_t), \\
C^3_t &= DB_3(s^3_f \otimes C^3_f + s^3_b \otimes C^3_b + b^3_t).
\end{aligned}
$$

Here \(s^i_f\) and \(s^i_b\) are learned forward and backward reference weights, \(b^i_t\) is a learned bias or supplementary term for the current frame, \(\otimes\) denotes element-wise multiplication, and \(DB_i\) is a depth block at scale \(i\) [2511.08938].

The bias term is central to the paper’s definition of harmonization. It allows the model to rely on the current frame’s own features when neither reference is reliable, when both references are weak, or when the target region is not well supported by either reference. Accordingly, BRHVC does not reduce harmonization to choosing the better of two references; it also models cases in which both references should be downweighted [2511.08938].

At the decoder, \(x_t\) is unavailable, so the method uses \(\hat{y}_t\) together with pixel-feature alignment (PFA) to guide decoder-side fusion:

$$
C'^1_t, C'^2_t, C'^3_t = BCF_D(\{C^1_f,C^1_b\}, \{C^2_f,C^2_b\}, \{C^3_f,C^3_b\} \mid \{\hat{y}_t, x^{warp}_{t,f}, x^{warp}_{t,b}\}).
$$

The paper also states that BCF estimates reference weights from pixel-domain similarity between the current frame and the warped forward and backward references. Its visualizations show that, for large spans, the more useful reference receives larger average weight, the bias term becomes larger when compensation is harder, and for small spans the two references become more similar in importance while the bias shrinks [2511.08938].

## 5. Optimization, datasets, and reported compression performance

BRHVC is built on the backbone ideas of **DCVC-B** and **DCVC-DC**, while replacing single-direction contexts with **bi-directional reference contexts**. Its encoder/decoder architecture is described as largely like DCVC-DC, with the principal difference being the use of bidirectional contexts [2511.08938].

The reported training setup is specific. BRHVC is trained on **Vimeo-90k** with 7-frame sequences, fine-tuned on original Vimeo videos with 17-frame sequences, and tested on **HEVC Classes B–E**, **UVG**, and **MCL-JCV**. The main testing configuration uses **IP = 32**. The optimizer is **AdamW**, the batch size is **8**, the random crop size is **256×256**, random sequence reversal is applied with probability **0.5**, and the training strategy is a **multi-stage training strategy following DCVC-B** [2511.08938].

The main reported compression results are on HEVC datasets. BRHVC achieves **27.3%** average bitrate saving over **VTM-LDB**, **32.0%** average bitrate saving on HEVC datasets, and surpasses **VTM-RA**, which gets **30.9%** average saving on HEVC datasets. The reported gain is especially strong on **HEVC Class D**, where BRHVC saves **44.7%**. The paper also states that BRHVC outperforms the previous state-of-the-art neural methods **DCVC-DC**, **DCVC-FM**, and **DCVC-B** [2511.08938].

Under MS-SSIM optimization, the method also surpasses DCVC-B, with average **-44.0%** versus DCVC-B’s **-42.4%** on HEVC B–E. The paper further reports that BRHVC’s advantage over DCVC-B increases as IP increases: **IP8: -15.3% w/o I-frame**, **IP16: -17.8%**, and **IP32: -21.6%**. This is presented as direct support for the claim that URC becomes more severe as frame span grows and that BRHVC is specifically effective in that regime [2511.08938].

The paper also notes an important limitation: BRHVC is **less strong on high frame-rate datasets** such as UVG and MCL-JCV, and it explicitly states that further study is needed there. That caveat constrains the generality of its strongest performance claims, which are centered on the HEVC datasets under random-access conditions [2511.08938].

## 6. Ablation evidence, interpretive scope, and related misconceptions

The ablation results isolate the contributions of the two core modules. Starting from a **Baseline (BL)** that is described as a reference implementation close to DCVC-B, the paper reports that **BL + BMC** improves by **6.4%**, **BL + BCF** improves by **6.6%**, and **BL + BCF + BMC** improves by **12.3%**. These figures are used to argue that BCF directly addresses URC, BMC strengthens motion estimation, and the two are complementary when combined [2511.08938].

A separate multi-scale flow ablation is reported for BMC. Replacing the largest-scale flows causes a **6.7% BD-rate loss**, replacing the intermediate-scale flows causes **4.2%**, and replacing the smallest-scale flows causes **3.0%**. The paper interprets this as evidence that all scales contribute and that larger-scale flows are especially important for difficult large-span motion. Visualizations are said to show that BMC helps avoid disordered estimated motion in background regions when the span is large [2511.08938].

A common misconception is to treat BRHVC as merely another bidirectional concatenation model. The paper argues against that view directly. BCF does not simply concatenate forward and backward contexts; it explicitly predicts \(s^i_f\) and \(s^i_b\), and it supplements them with \(b^i_t\). This means that harmonization in BRHVC is not equivalent to equal-weight averaging, nor is it limited to binary selection of one reference over the other. The current-frame supplementary term is part of the method’s formal mechanism [2511.08938].

Another misconception is to treat BRHVC as isolated from earlier learned hierarchical B-frame codecs. The related literature shows a clear progression from pixel-domain bidirectional warping and mask-based fusion with flexible-rate gain control to feature-level deformable alignment and conditional coding, and then to explicit motion-guided reference harmonization under URC. This suggests that BRHVC is best read as a specialized development within learned hierarchical bi-directional video compression rather than as a departure from that line of work [2206.13613][2306.16544].

Taken together, the evidence supports a specific characterization: BRHVC is a neural B-frame codec whose novelty lies in making the unequal usefulness of two references the primary modeling target. BMC improves the motion representation and motion compensation; BCF uses that improved compensation to weight reference contexts adaptively; and the resulting system achieves state-of-the-art compression performance on the reported HEVC random-access benchmarks, including surpassing **VTM-RA on HEVC datasets** [2511.08938].

Source: https://www.emergentmind.com/topics/bi-directional-reference-harmonization-video-compression-brhvc