Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment

Abstract: The rapid advancement of generative models has led to a growing volume of AI-generated videos, making the automatic quality assessment of such videos increasingly important. Existing AI-generated content video quality assessment (AIGC-VQA) methods typically estimate visual quality by analyzing each video independently, ignoring potential relationships among videos. In this work, we revisit AIGC-VQA from an inter-video perspective and formulate it as a reference-aware evaluation problem. Through this formulation, quality assessment is guided not only by intrinsic video characteristics but also by comparisons with related videos, which is more consistent with human perception. To validate its effectiveness, we propose Reference-aware Video Quality Assessment (RefVQA), which utilizes a query-centered reference graph to organize semantically related samples and performs graph-guided difference aggregation from the reference nodes to the query node. Experiments on existing datasets demonstrate that our proposed RefVQA outperforms state-of-the-art methods across multiple quality dimensions, with strong generalization ability validated by cross-dataset evaluation. These results highlight the effectiveness of the proposed reference-based formulation and suggest its potential to advance AIGC-VQA.

• The paper proposes a reference-aware modeling framework (RefVQA) that reframes video quality assessment from isolated regression to comparative reasoning using semantic retrieval.
• The methodology utilizes dual visual and alignment branches to capture feature discrepancies, achieving state-of-the-art performance on multiple benchmarks.
• Experimental evaluations demonstrate improved correlation metrics and efficiency, highlighting potential extensions to other generative content evaluations.

Reference-Aware Comparative Modeling for AI-Generated Video Quality Assessment

Introduction

The unprecedented progress in generative models has led to an explosion of AI-generated videos (AIGVs) across creative and industrial domains, emphasizing the critical need for robust automatic video quality assessment (VQA) systems. However, the unique multi-dimensional artifacts in AIGVs—such as structural incoherence, motion temporal artifacts, and prompt misalignment—have exposed the limitations of conventional VQA methodologies, which typically operate under an isolated, sample-wise quality regression paradigm. This paper addresses these shortcomings by reframing AIGC-VQA as a reference-aware, inter-video comparative modeling problem. The authors introduce RefVQA, a framework that capitalizes on semantically anchored reference retrieval and graph-guided difference aggregation, in both visual and cross-modal alignment spaces, to predict perceptual video quality with greater alignment to human subjective judgments (2604.17074).

The motivation and theoretical premise underlying this reformulation are illustrated below.

Figure 1: Motivation and formulation of the reference-aware AIGC-VQA framework, highlighting the transition from isolated assessment to context-sensitive comparative reasoning.

Reference-Aware Comparative Modeling: The RefVQA Framework

Traditional AIGC-VQA approaches, even those incorporating multi-branch architectures or multi-modal transformers, conduct quality assessment in isolation, neglecting the inherently comparative disposition of human perception. RefVQA pivots from this paradigm by integrating relative assessment: the query video is evaluated in the context of semantically comparable reference samples retrieved using prompt similarity via Sentence-BERT embeddings.

The framework is structured as follows:

1. Reference Retrieval and Graph Construction: The system retrieves videos with highly similar prompts to the query, leveraging prompt semantics as a robust, generation-quality-invariant anchor. These samples are organized into a star-shaped reference graph centered on the query node, with edge weights proportional to prompt similarity.
2. Visual Branch: This branch encodes spatio-temporal features using a Swin Transformer, followed by 3D pooling. Rather than direct reference feature aggregation, node-wise differences (query minus reference) are computed, aggregated with similarity weighting, transformed with learnable projections, and gated adaptively before concatenation with the intrinsic query features.
3. Alignment Branch: Using BLIP-based representations, this branch operates analogously to capture video-prompt semantic alignment discrepancies via difference aggregation over the reference graph.
4. Quality Prediction: The reference-enhanced visual and alignment representations are fused and input to a lightweight regressor for final perceptual quality prediction.

   Figure 2: The RefVQA pipeline, including query-centered graph construction, dual-branch reference-aware difference aggregation, and feature fusion for quality prediction.

RefVQA contrasts with preceding pairwise or rank-loss approaches, which impose relational constraints solely at optimization; here, inter-video relationships directly shape the learned representations within both visual and alignment modalities.

Experimental Evaluation

Datasets and Protocol

Evaluation is performed across three benchmarks: T2VQA-DB, LGVQ, and FETV, covering diverse generative model outputs, annotation schemes (MOS and dimensional labels), and varied prompt semantics. Metrics include PLCC, SRCC, KRCC, and RMSE, enabling both absolute and rank-based performance analyses.

Numerical Results and Analysis

RefVQA achieves the strongest reported performance across all metrics and datasets:

• T2VQA-DB: SRCC = 0.826, PLCC = 0.835, KRCC = 0.642, RMSE = 8.429. Substantial improvements are observed over recent state-of-the-art methods such as VE-Bench QA and T2VEval, especially in ranking-based correlations.
• LGVQ: RefVQA leads spatial, temporal, and alignment dimensions, with alignment SRCC/PLCC up to 0.731/0.738—demonstrating the framework’s ability to leverage semantic references beyond visual facets.
• FETV Cross-Dataset Generalization: When trained on LGVQ, RefVQA surpasses all baselines on FETV, with a marked gain on the alignment and temporal metrics (SRCC temporal: 0.719 vs UGVQ’s 0.512).

Explicit comparative modeling of reference discrepancies, especially in the alignment branch, yields the largest gains; ablation studies confirm the necessity and synergy of both branches.

Qualitative and Analytical Insights

Visual exploration reinforces the benefit of reference-aware modeling. In scenarios with complex artifacts (e.g. semantic incompleteness, motion jitter, or blurring), direct comparison with relevant references refines both the ranking and absolute score predictions.

Figure 3: Qualitative comparison of predicted quality with and without reference-aware modeling, highlighting superior granularity and reliability in the proposed method.

Reference-enhanced predictions show reduced dispersion from ground-truth MOS, as illustrated in score-ground truth scatter plots.

Figure 4: Tightened score-ground truth MOS correlation under reference-aware modeling.

Alignment and discriminability of the learned representations are also improved, as low- and high-MOS clusters become better separated in the feature space.

Qualitative comparisons under varied distortion contexts (spatial, temporal, alignment) further reveal RefVQA’s sensitivity and adaptability that are otherwise lacking in reference-free systems.

Figure 5: Qualitative comparison of predicted quality scores under different types of distortion.

Ablation and Efficiency

Ablations indicate:

• Superior performance is robust to the choice of similarity thresholds, retrieval pool size, and feature aggregation schemes, with difference-based, graph-weighted aggregation outperforming naive averaging/concatenation.
• Both the dual-branch configuration and similarity-weighted reference retrieval are critical; prompt-based retrieval surpasses feature-based or random selection.
• Training efficiency is optimized via staged training with backbone freezing in later phases: RefVQA achieves lower inference time (99 ms/sample) compared to T2VQA and AIGVEval at substantially higher accuracy.

Limitations and Future Directions

While prompt-based reference retrieval provides situationally robust semantic anchors, it may be insufficient in cases of severe prompt-video drift or quality-factor divergences. Integrating quality-aware retrieval (potentially via multimodal embeddings) represents a promising direction. Further, more expressive relational architectures—capturing inter-reference relationships (e.g., via full-graph GNNs, set-level attention, or ordinal graphs)—could harness richer comparative signals. Adopting set-wise or subset-based training, rather than repeated query-centric evaluation, may enhance computational efficiency and model generalization.

Theoretical and Practical Implications

The reference-aware paradigm aligns computational VQA with fundamental psychoperceptual concepts—quality perception is relational and context-sensitive, not absolute. RefVQA’s approach can be systematically extended to other generative evaluation tasks (e.g., text-to-image, video editing QA), as well as to interactive human-AI preference modeling. Methodologically, the integration of semantic retrieval, structured relational modeling, and difference reasoning may fuel more robust and generalizable quality and alignment assessment systems, especially critical as generative models proliferate in high-stake, user-facing applications.

Conclusion

This work reframes AIGC-VQA from an isolated regression task to a semantically entangled, reference-aware comparative reasoning challenge. Through efficient prompt-based reference retrieval and graph-guided feature difference integration in both visual and alignment modalities, RefVQA delivers state-of-the-art, generalizable video quality prediction with strong empirical and qualitative support. The reference-aware hypothesis and underlying model architectures pose a significant foundation for future explorations in generative content quality assessment and wider, relationally grounded multimodal analysis.