A Mamba-based Perceptual Loss Function for Learning-based UGC Transcoding

Abstract: In user-generated content (UGC) transcoding, source videos typically suffer various degradations due to prior compression, editing, or suboptimal capture conditions. Consequently, existing video compression paradigms that solely optimize for fidelity relative to the reference become suboptimal, as they force the codec to replicate the inherent artifacts of the non-pristine source. To address this, we propose a novel perceptually inspired loss function for learning-based UGC video transcoding that redefines the role of the reference video, shifting it from a ground-truth pixel anchor to an informative contextual guide. Specifically, we train a lightweight neural quality model based on a Selective Structured State-Space Model (Mamba) optimized using a weakly-supervised Siamese ranking strategy. The proposed model is then integrated into the rate-distortion optimization (RDO) process of two neural video codecs (DCVC and HiNeRV) as a loss function, aiming to generate reconstructed content with improved perceptual quality. Our experiments demonstrate that this framework achieves substantial coding gains over both autoencoder and implicit neural representation-based baselines, with 8.46% and 12.89% BD-rate savings, respectively.

• The paper introduces PT-Loss, a Mamba-based perceptual loss trained with weakly supervised Siamese ranking to treat compressed UGC references as context rather than pixel-perfect targets.
• Integrating PT-Loss into DCVC and HiNeRV reduces source-based BD-rate by 8.46% and 12.89% in PSNR, respectively, with the largest gains on heavily degraded references.
• The method achieves these improvements with lower complexity than a Swin-based alternative, although its results rely on VMAF proxy labels, conventional-codec training data, and evaluation on 15 sequences.

This paper proposes PT-Loss, a perceptually motivated loss function for learning-based user-generated content (UGC) video transcoding, built on a lightweight Mamba-based quality assessment network trained with weakly supervised Siamese ranking. The central premise is that UGC transcoding differs fundamentally from conventional coding: the input reference is already degraded by prior compression, so fidelity-driven rate-distortion optimization (RDO) forces codecs to reproduce pre-existing artifacts rather than maximize perceived quality of the delivered content.

Motivation and problem formulation

In the standard UGC delivery pipeline, a pristine source $\mathbf{S}$ captured on a mobile device is compressed on-device into a non-pristine reference $\mathbf{R}$, uploaded, and then transcoded by the platform into a distorted video $\mathbf{D}$. Conventional and learning-based codecs alike optimize distortion against $\mathbf{R}$ using metrics such as PSNR or SSIM, which is appropriate only when the reference is pristine. For UGC, this "fidelity trap" causes the codec to preserve visible compression artifacts, noise, and editing flaws.

The proposed remedy redefines the role of the reference: instead of a pixel-level ground truth, $\mathbf{R}$ serves as semantic context, while the optimization target becomes estimated perceptual degradation relative to the (unavailable) pristine source. This reframing follows from prior work on full-reference VQA for UGC transcoding (Qi et al., 2024) and ranking-inspired training [feng2024rankdvqa].

Network architecture

The quality model processes co-located spatio-temporal patches of $256 \times 256 \times 12$ from both $\mathbf{R}$ and $\mathbf{D}$. The key architectural choice is replacing the Swin Transformer backbone used in RankDVQA-UGC with a Selective Structured State-Space Model (Mamba) [gu2024mamba], decoupling spatial feature extraction from temporal selective scanning. This yields linear complexity $O(L)$ in sequence length while capturing long-range temporal dependencies — important for distinguishing temporal consistency from flickering artifacts — at substantially lower inference cost than attention-based designs such as VideoMamba [li2024videomamba]. Multi-level features are fused to produce a patch-level quality index, aggregated via a differentiable pooling layer into a deterministic continuous clip-level score, ensuring gradient stability when backpropagated through the codec.

Weakly supervised training

Because large-scale human-annotated UGC transcoding data do not exist, the authors adopt a proxy-label strategy: 252 pristine sources (from BVI-DVC, CLIC 2022, YouTube-UGC 2K, plus six self-captured sequences) are first compressed with x264 at QP = 30/37/42 to form non-pristine references, then transcoded with x264, AV1, and VP9 across wide QP ranges. Quality labels for each $(\mathbf{R}, \mathbf{D})$ pair are computed as VMAF between $\mathbf{R}$0 and its pristine source $\mathbf{R}$1 — not the reference — effectively distilling source-aware knowledge from a full-reference metric into a network that sees only the non-pristine reference. This produces 604,800 patch pairs trained via Siamese ranking, with an 8:2 single-source/cross-source sampling ratio; after convergence, fine-tuning uses single-source pairs only, since same-content quality discrimination matters most for loss behavior. A stated limitation: training material was generated exclusively with conventional codecs for speed; neural-codec-generated training data might improve results but is left as future work.

Codec integration

The loss is integrated into two architecturally distinct neural codecs:

• DCVC (autoencoder-based) [li2021deep]: only the P-frame model is fine-tuned on Vimeo-90k, with total loss $\mathbf{R}$2, where $\mathbf{R}$3 magnitudes are adaptively aligned to MSE over the first five iterations.
• HiNeRV (INR-based, per-video overfitting) [kwan2023hinerv]: MS-SSIM in the original loss is replaced by PT-Loss, giving $\mathbf{R}$4.

In both cases $\mathbf{R}$5, validated by an ablation over $\mathbf{R}$6 showing optimal performance at 0.2.

Results

Evaluation uses a 15-sequence subset of BVI-UGC (one representative per UGC category), with three reference-quality groups (x264 QP 30/37/42). Crucially, quality is measured against the uncompressed source rather than the non-pristine reference.

Codec	Overall BD-rate (PSNR)	Overall BD-rate (VMAF)	Complexity overhead
DCVC + PT-Loss	−8.46%	−8.64%	none (offline training loss)
HiNeRV + PT-Loss	−12.89%	−10.83%	+21.3% enc. time, +0.8M params, +1.4G MACs

Gains are largest for low-quality references (e.g., −12.98% PSNR BD-rate for DCVC, −16.15% for HiNeRV), consistent with the motivation that fidelity-driven coding is most harmful when the input is heavily degraded. HiNeRV benefits more than DCVC, which the authors attribute to the overfitting nature of INR codecs. Against the Swin Transformer-based RankDVQA-UGC loss, PT-Loss delivers higher overall savings with roughly one-quarter the MACs (1.4G vs. 5.2G) and one-sixth the parameters (0.8M vs. 4.6M), confirming the efficiency motivation for the Mamba design. Qualitative comparisons show improved rendering of text and structured edges at matched bitrates, corroborated by higher PSNR/VMAF against the source.

Limitations and open questions

Several caveats bear directly on the reported results. First, evaluation covers only 15 of BVI-UGC's 60 sequences, selected for feature coverage; broader validation remains open. Second, training labels derive from VMAF against pristine sources rather than subjective scores — reliable in prior ranking-based studies, but an assumption nonetheless. Third, training simulations use only conventional codecs (x264, AV1, VP9), so generalization to references produced by neural codecs is unverified. Fourth, the gains depend on the hyperparameter $\mathbf{R}$7 and the magnitude-alignment heuristic, whose sensitivity beyond the tested range is not characterized. Finally, the method still operates within fidelity-oriented codec architectures; whether perceptual losses alone can enable reconstructions exceeding reference quality, or whether generative compression designs are required, is explicitly left open by the authors.

Conclusion

The paper demonstrates that treating the non-pristine reference as contextual guidance rather than a pixel anchor yields consistent BD-rate savings (up to 12.89%) across autoencoder- and INR-based neural codecs, with the largest gains on low-quality inputs, and does so with a Mamba-based loss network whose complexity is markedly lower than transformer alternatives. The approach is model-agnostic and adds no decoding cost, though its reliance on proxy labels, conventional-codec training data, and a modest test subset temper the generality of the conclusions.