MBCodec: Advances in Neural Audio & Video Coding

• MBCodec is a neural video and audio coding framework that employs hierarchical codebook design and adaptive fusion to achieve fine-grained compression and robust semantic-acoustic disentanglement.
• Its audio pipeline uses multi-codebook residual vector quantization with semantic tokenization and PQMF integration to deliver near-lossless speech reconstruction at extremely low bitrates.
• The video pipeline features an interactive dual-branch motion auto-encoder and selective temporal fusion, yielding up to 38% BD-rate reductions and significant energy savings in multi-codec streaming.

MBCodec is a research-driven framework encompassing recent advances in neural video and audio coding, specifically targeting fine-grained compression, robust semantic-acoustic disentanglement, and energy-efficient multi-codec workflows. The MBCodec approach is characterized by domain-specific technical innovations for both video (particularly B-frame compression) and high-fidelity audio codecs. This article surveys MBCodec’s underlying principles, architecture, performance metrics, and research developments as presented in recent arXiv publications.

1. Foundations and Motivation

MBCodec is motivated by two central challenges:

• The limitations of traditional and neural codecs in capturing and reconstructing rich semantic and acoustic signals at extremely low bitrates.
• The increasing energy costs and operational burdens of multi-codec workflows in adaptive streaming environments.

Prior codecs either relied on single-codebook vector quantization or repurposed tools designed primarily for P-frame video compression, resulting in poor disentanglement of semantic/acoustic content, suboptimal rate-distortion performance, and redundancy across multi-codec bitrate ladders. MBCodec (as defined for both audio and video coding domains) employs new hierarchical and interaction-based modeling schemes to address these deficits (Zhang et al., 21 Sep 2025, Sheng et al., 9 Jun 2025, Menon et al., 2023).

2. Technical Design for Audio Compression

The audio variant of MBCodec utilizes a multi-codebook Residual Vector Quantization (RVQ) architecture. Key elements include:

• Semantic-Acoustic Disentanglement:

MBCodec builds separate pathways for semantic and acoustic processing. Self-supervised semantic tokenization, provided by models like HuBERT, guides the vector quantization module, ensuring the preservation of linguistic content. Concurrently, a multi-channel Pseudo-Quadrature Mirror Filter (PQMF) splits raw audio into frequency subbands, supervising the RVQ codebooks so that finer acoustic details are captured hierarchically.

• RVQ Hierarchy and Codebook Multiplicity:

The RVQ process sequentially quantizes residuals at each layer, allowing the representational capacity to scale multiplicatively as $N^M$ (where $N$ is codebook size; $M$ is the number of codebooks). This configuration surpasses additive single-codebook systems in efficiency, enabling bitrates as low as 2.2 kbps for 24 kHz audio at 170× compression.

• Adaptive Dropout in Training:

Recognizing that deeper RVQ layers convey diminishing residual information, MBCodec applies non-uniform (e.g., exponential decay or half-Gaussian) sampling to differentially weight dropout depths, thus emphasizing learning in the crucial leading layers.

• Loss Functions and PQMF Integration:

Training leverages a composite loss: adversarial ($L_{GAN}$), reconstruction ($L_{recon}$), and VQ losses ($L_{vq}$), along with semantic and acoustic guidance terms:

$$L_{seman} = \cos(Q_{seman}, S) \ L_{n(acous)} = \cos(A_1 Q_{acous}^{(d)}, A_2 H_i^{(d)}) \ L_{total} = L_{GAN} + L_{recon} + L_{vq} + L_{seman} + \sum_n L_{n(acous)}$$

where $Q_{seman}$ and $Q_{acous}$ denote quantized latent representations.

This configuration results in near-lossless speech reconstruction, high-fidelity reproduction of both semantic and acoustic features, and substantial reduction in bitrates and computational overhead.

3. Technical Advances for Neural B-Frame Video Coding

MBCodec’s video pipeline incorporates fine-grained motion compression and selective temporal fusion tailored for neural B-frame coding:

• Interactive Dual-Branch Motion Auto-Encoder:

Separately processes forward and backward motion vector differences (MVDs), capturing asymmetric temporal correlations. The motion information interaction (MII) module enables cross-branch feature exchange, governed by query/key/value transformations and attention mechanisms:

$$Q_{t \rightarrow f}, K_{t \rightarrow f}, V_{t \rightarrow f} = k_f(M_{t \rightarrow f}^i) \ \tilde{M}_{t \rightarrow f} = \left[\operatorname{softmax}(K_{t \rightarrow b}) (V_{t \rightarrow b})^T\right]^T \operatorname{softmax}(Q_{t \rightarrow f})$$

• Per-Branch Adaptive Quantization:

Each branch (forward/backward) receives independently learned quantization steps, formed by global and channel-wise components:

$$q_{t \rightarrow f}^{(m,e)} = q_{t \rightarrow f}^{(m,e,global)} \odot q_{t \rightarrow f}^{(m,e,ch)}$$

• Interactive Motion Entropy Modeling:

Post-quantization, motion latents are partitioned, and an entropy model exploits directional priors from both forward and backward branches for efficient coding:

$$p(\hat{m}_{t \rightarrow f}^{i} | z_t^m, o_t^m, \hat{m}_{t \rightarrow f}^{<i}, \hat{m}_{t \rightarrow b}^{<i}) \sim \mathcal{L}(\mu_{t \rightarrow f}^{(m,i)}, \sigma_{t \rightarrow f}^{(m,i)})$$

• Selective Temporal Fusion and Hyperprior Alignment:

Fusion weights (bi-directional, multi-scale) are adaptively computed for each region, enabling discriminative utilization of temporal contexts. Contextual entropy modeling is reinforced by a hyperprior-based implicit alignment mechanism, realigning temporal priors via depth-wise convolution and attention:

$$Q_t^\gamma = s_z(z_t^\gamma) \ K_t^\gamma, V_t^\gamma = s_o(o_t^\gamma) \ \bar{o}_t^\gamma = \left[\operatorname{softmax}(K_t^\gamma) (V_t^\gamma)^T\right]^T \operatorname{softmax}(Q_t^\gamma) \ \tilde{o}_t^\gamma = o_t^\gamma + j_o(\bar{o}_t^\gamma \odot s_o(o_t^\gamma))$$

Ablation results confirm each module’s incremental contribution to coding efficiency.

4. Multi-Codec Bitrate-Ladder Estimation and Energy Efficiency

In the adaptive video streaming context, MBCodec operationalizes the Multi-Codec Bitrate-Ladder Estimation (MCBE) scheme. MCBE streamlines the encoding, storage, and transmission process when multiple codecs are supported (e.g., AVC, HEVC, AV1), as detailed in (Menon et al., 2023):

• Bitrate Ladder Consolidation:

MCBE removes representations from newer codecs (HEVC, AV1) if they fall below the AVC rate-distortion (RD) curve, eliminating redundant and underperforming streams.

• Perceptual Redundancy Management:

A Just Noticeable Difference (JND) threshold, measured in VMAF points, is applied to prune quality representations that do not yield perceptually significant improvement.

• Random Forest-Based VMAF Prediction:

For each codec and resolution, random forest regression models (hyperparameters: min_samples_leaf=1, min_samples_split=2, n_estimators=100, max_depth=14) infer VMAF scores from low-complexity spatio-temporal features.

• Workflow and Algorithm:

Algorithmically, MCBE prunes representations exceeding a perceptual v_max threshold or falling under the JND gap v_J. The remaining representations undergo geometric comparison across RD curves.

• Quantitative Energy Savings:

In live streaming with AVC/HEVC/AV1 client support and JND=6 VMAF points, MCBE yields up to 56.45% reduction in encoding energy, 94.99% reduction in storage energy, and 77.61% reduction in transmission energy, compared to the industry-standard baseline.

5. Empirical Performance and Evaluation

MBCodec demonstrates competitive and, in many scenarios, superior performance to the prevailing state-of-the-art codecs:

• Audio:

Objective metrics (PESQ, SI-SDR, STFT/Mel distance, MUSHRA, WER, semantic similarity) consistently favor MBCodec over DAC, EnCodec, and SpeechTokenizer. Best results are achieved with 16 codebooks at 50 Hz.

• Video:

Rate-distortion evaluations (PSNR, BD-rate) on benchmark datasets (MCL-JCV, UVG, HEVC sequences) reveal average BD-rate reductions of up to 38.0% versus the HM-RA-GOP16 anchor, and parity or superiority w.r.t. H.266/VVC (VTM) in random-access settings. Selective temporal fusion and motion interaction modules deliver measurable gains.

• Streaming Efficiency:

In live streaming, MCBE maintains negligible processing latency (0.37 s for a 4 s 2160p segment) while dramatically reducing operational energy consumption.

6. Implications and Future Directions

MBCodec’s explicit semantic-acoustic disentanglement, hierarchical codebook design, and adaptive fusion/quantization mechanisms set a new technical standard for neural coding. Immediate implications include:

• Efficient deployment in real-time speech synthesis and transmission systems, maximizing fidelity at low bitrates.
• Scalable video coding with enhanced energy efficiency and reduced redundancy in multi-codec streaming environments.
• Modular architecture enabling further research on adaptive dropout distributions, disentanglement strategies, and application to multimodal compression frameworks.

A plausible implication is broader adoption in cloud-based conferencing and collaborative media platforms, where codec agility and resource minimization are critical.

7. Relation to Contemporary Research and Contextual Significance

MBCodec synthesizes contributions from both neural audio and video coding research efforts. Its design leverages recent advances in SSL-based tokenization, hierarchical quantization, and spatio-temporal feature exploitation, aligning with ongoing trends favoring learned, data-driven, and energy-conscious compression architectures.

In summary, MBCodec defines a comprehensive, technically rigorous framework for advanced neural coding in speech and video domains, achieving substantial improvements in compression efficiency, fidelity, and operational resource utilization through its multifaceted innovations.