Q-CLIP: Vision-Language VQA Framework

• Q-CLIP is a no-reference video quality assessment method that leverages a frozen, video-aware CLIP-style backbone along with five learnable quality prompts to predict perceptual quality.
• It converts sampled video frame features and quality-level text prompts into cosine similarity scores, forming a 5-dimensional quality distribution for regression.
• Its compact adaptation design, including a Shared Cross-Modal Adapter with only 0.14M trainable parameters, offers significant computational savings over conventional fine-tuning pipelines.

Q-CLIP is a no-reference video quality assessment (VQA) method that adapts a CLIP-like vision-LLM to predict perceptual quality in in-the-wild user-generated videos. It is presented as the first fully VLM-based framework for VQA, and its core design couples a frozen video-aware CLIP-style backbone with a lightweight Shared Cross-Modal Adapter (SCMA), five learnable quality-level prompts, and a cosine-similarity-based quality prediction mechanism. The method is explicitly motivated by the claim that video quality depends not only on semantics but also on distortion, motion, aesthetics, and perceptual consistency, and that classification-style pretraining transfers only part of that structure to VQA (Mi et al., 8 Aug 2025).

1. Problem setting and conceptual motivation

Q-CLIP addresses no-reference VQA for in-the-wild user-generated videos, where the input is a video without a pristine reference and the target is a perceptual quality score such as MOS or DMOS. The paper characterizes this setting as intrinsically multi-factorial: video quality depends on semantics, distortion, motion, aesthetics, and perceptual consistency, rather than on semantic discrimination alone. On that basis, it argues that mainstream VQA pipelines built around large visual backbones pretrained on classification datasets such as ImageNet or Kinetics-400 are limited in two ways: transferring only semantic knowledge is insufficient, and large-scale pretraining is computationally expensive (Mi et al., 8 Aug 2025).

The computational argument is made concretely. The paper states that pretraining on large datasets can cost one to two orders of magnitude more than training directly on VQA data, and cites FAST-VQA as an example: pretraining on Kinetics-400 is about $10\times$ and $200\times$ more expensive than fine-tuning on LSVQ and KoNViD-1k, respectively. The proposed alternative is to begin from a vision-LLM, on the premise that VLMs encode broader multimodal concepts than classification models and may therefore better cover the ingredients of perceptual quality (Mi et al., 8 Aug 2025).

This suggests a shift in problem formulation. Rather than treating VQA as pure scalar regression on visual features, Q-CLIP treats it as a cross-modal alignment problem in which video features are compared against a structured set of textual quality anchors. A plausible implication is that the method inherits CLIP’s semantic priors while explicitly attempting to reshape them toward graded quality perception.

2. System architecture and prediction pipeline

The high-level pipeline of Q-CLIP is compact. Frames are sampled from a video, the sampled frames are encoded through a frozen CLIP-like visual branch, quality-level text prompts are encoded through the frozen text branch, both modalities are adapted by SCMA, cosine similarities are computed between the video representation and five quality prompts, and those similarities are converted into a continuous quality prediction via a learned weighted sum (Mi et al., 8 Aug 2025).

The backbone is not the original CLIP model. Q-CLIP uses PE$_{core}$-L from the “Perception Encoder” work, which the paper describes as a CLIP variant pre-tuned on video data. This choice is motivated by a specific limitation of standard VLMs: they are mostly trained on static images and are weak at temporal modeling. Accordingly, the backbone is intended to provide a video-aware starting point while preserving the CLIP-style dual-encoder structure (Mi et al., 8 Aug 2025).

The paper defines the output representations explicitly. For each video, Q-CLIP produces one video representation $V$ and five text representations

$$T = \{t_{exc}, t_{good}, t_{fair}, t_{poor}, t_{bad}\}.$$

Cosine similarities between $V$ and these five prompt features form a 5-dimensional “quality-level-related distribution,” which is then mapped to a scalar quality score. The backbone remains frozen throughout this process. The paper repeatedly emphasizes that adaptation is achieved without full backbone fine-tuning, which is central to both its efficiency claim and its positioning relative to prior VQA systems (Mi et al., 8 Aug 2025).

Several misconceptions are precluded by this formulation. Q-CLIP is not a full-reference metric, not a general image IQA model, and not a re-training of the CLIP backbone. It is specifically a no-reference VQA system whose adaptation locus is deliberately narrow.

3. Shared Cross-Modal Adapter (SCMA)

SCMA is the main technical contribution of Q-CLIP. Its stated purpose is to refine both visual and textual representations in a coordinated way, addressing two problems the paper identifies in downstream VQA adaptation of VLMs: insufficient intra-modal representational capacity for fine-grained perceptual differences, and a modality gap between visual and textual features. SCMA is “cross-modal” because it is applied to both branches, and “shared” because the same adapter architecture is reused across them (Mi et al., 8 Aug 2025).

The paper distinguishes two forms of the module: E-SCMA, inserted around encoder layers, and P-SCMA, inserted around projection layers. Let the inputs to visual and text encoders at layer $k$ be $V_k$ and $T_k$, and let the frozen backbone encoder layers be $E_k^v$ and $200\times$0. The frozen encoders first produce

$200\times$1

E-SCMA then computes residuals

$200\times$2

$200\times$3

After normalization,

$200\times$4

the adapted representations are formed as

$200\times$5

Here $200\times$6 is a linear layer for dimensionality reduction, $200\times$7 is a feed-forward network, $200\times$8 is a linear layer for dimensionality expansion, and $200\times$9 are learnable scalars. For P-SCMA, the paper states that the processing flow is similar, except that the FFN is replaced by a single linear layer (Mi et al., 8 Aug 2025).

The sharing operates at two levels. There is branch sharing, since the same SCMA design is used for both visual and text pathways, and inter-layer sharing, since parameters are shared across layers inside the E-SCMA feed-forward network to avoid parameter growth and overfitting. The paper quantifies the entire trainable adaptation as only 0.14M trainable parameters and reports reductions of $_{core}$0, $_{core}$1, $_{core}$2, and $_{core}$3 relative to MBVQA, CLIPVQA, Q-Align, and FAST-VQA, respectively (Mi et al., 8 Aug 2025).

This parameterization clarifies the intended operating regime. Q-CLIP is not attempting to relearn video-quality semantics from scratch; it is attempting to bias a frozen VLM toward quality-sensitive alignment while leaving the underlying multimodal prior largely intact. A plausible implication is that the shared design regularizes the two branches toward a common quality geometry rather than permitting them to drift independently.

4. Quality-level prompting and cross-modal quality representation

The second central component is a set of five learnable quality-level prompts. The paper argues that binary antonym prompts such as “good” versus “bad” are too coarse for fine-grained quality prediction, and instead discretizes quality into five ordered levels: excellent, good, fair, poor, and bad. The base textual template is

$_{core}$4

where $_{core}$5 (Mi et al., 8 Aug 2025).

To permit limited adaptation without rewriting the prompt semantics entirely, Q-CLIP prepends learnable tokens: $_{core}$6 The resulting prompt set is

$_{core}$7

These prompts are not merely labels. They define a structured quality axis against which each video is compared. The paper states that the model computes cosine similarities between the video feature and all five quality anchors, producing the “quality-level-related distribution” used for prediction (Mi et al., 8 Aug 2025).

A notable methodological nuance appears here. The paper repeatedly states that the SCMA is the only component that requires training, yet it also describes the prompts as learnable. This suggests that the frozen-backbone claim is strict, while the trainable adaptation may include both the adapter and prompt tokens. What is unambiguous is that the CLIP-like backbone itself remains frozen and that Q-CLIP does not rely on full-model fine-tuning (Mi et al., 8 Aug 2025).

The prompt construction also reveals the paper’s broader view of VQA. Quality is represented not as a raw scalar latent alone, but as similarity to a discretized set of human-interpretable quality concepts. This is closely related to a broader tendency in CLIP-based quality assessment to replace direct regression with language-anchored quality inference.

5. Sampling strategy, generalization, and empirical positioning

Beyond adapter design and prompting, Q-CLIP explicitly investigates frame sampling. The paper reports that different sampling strategies materially affect VQA performance, and that frame-difference-based sampling leads to better generalization performance across datasets. This is an important point because the model is otherwise designed around a mostly frozen backbone: improvements can arise not only from feature adaptation but also from how temporally informative frames are exposed to the VLM (Mi et al., 8 Aug 2025).

The paper states that extensive experiments demonstrate excellent performance on several VQA datasets. No numerical benchmark values are provided in the supplied text, but the framing is consistent: Q-CLIP is positioned as a computationally efficient alternative to heavy pretraining-and-fine-tuning pipelines, with empirical behavior that supports the use of VLM priors for VQA (Mi et al., 8 Aug 2025).

Several objective clarifications follow from the reported design. Q-CLIP is not a temporal-reasoning module added to a LLM; it is a VQA predictor built around a dual-encoder vision-language backbone. It is not a training-free method; rather, it is a training-light method whose core adaptation is intentionally small. It is also not simply a reuse of image-text cosine similarity, because its contribution lies in unified cross-modal adaptation through SCMA and in the use of ordered quality prompts rather than ad hoc text queries.

This suggests that Q-CLIP occupies an intermediate methodological position between fully supervised video-quality regressors and zero-shot CLIP-style prompt scoring. It preserves the latter’s cross-modal structure while introducing a compact supervised adaptation layer specific to VQA.

6. Relation to adjacent CLIP-based quality methods and terminological context

The name “Q-CLIP” sits within a broader family of CLIP-derived systems for quality assessment, but those systems differ substantially in task definition, supervision, and modality. In video, the closest antecedent is CLIPVQA, which is a blind in-the-wild VQA method that uses a CLIP-based Transformer, a MOS2Language encoder, and cross-attention between content and quality-language representations (Xing et al., 2024). In image quality assessment, QualiCLIP adapts CLIP for opinion-unaware no-reference IQA by fine-tuning only the image encoder through quality-aware image-text alignment on synthetically degraded images (Agnolucci et al., 2024), while BPCLIP combines a bottom-up multiscale image encoder with CLIP text embeddings derived from 40 image quality adjectives across six dimensions for both full-reference and no-reference IQA (Song et al., 22 Jun 2025). A distinct line is ML-CLIPSim, a full-reference machine-oriented quality metric built on a frozen CLIP visual encoder that aggregates intermediate patch-token similarities and global image embeddings rather than targeting human-perceptual IQA (Ding et al., 10 May 2026). Another related but methodologically different direction is MA-CLIP, which remains training-free and augments prompt-based CLIP-IQA with a magnitude-aware cue derived from the unnormalized image embedding (Liao et al., 13 Nov 2025).

Method	Setting	Distinguishing mechanism
Q-CLIP (Mi et al., 8 Aug 2025)	No-reference VQA	Frozen video-aware VLM, SCMA, five learnable quality prompts
CLIPVQA (Xing et al., 2024)	Blind in-the-wild VQA	Spatiotemporal Transformer with MOS2Language and cross-attention
QualiCLIP (Agnolucci et al., 2024)	NR-IQA	Quality-aware image-text alignment without MOS labels
BPCLIP (Song et al., 22 Jun 2025)	FR/NR IQA	Bottom-up multiscale cross-attention with quality-adjective text supervision
ML-CLIPSim (Ding et al., 10 May 2026)	Full-reference machine-oriented IQA	Frozen CLIP features with multi-layer patch-token and global similarity
MA-CLIP (Liao et al., 13 Nov 2025)	NR-IQA	Training-free fusion of cosine prompt alignment and embedding magnitude

A different source of confusion is QLIP, which is not a quality-assessment method at all, but a quadtree-based drop-in replacement for the CLIP vision encoder inside multimodal LLMs (2505.23004). That distinction matters because “Q-CLIP” can be informally used to refer either to quality-aware CLIP methods or to methods whose names merely begin with “Q.” In the present usage, Q-CLIP refers specifically to the VQA framework in (Mi et al., 8 Aug 2025).

Within this landscape, Q-CLIP is best understood as a video-domain specialization of the broader quality-aware CLIP trend. It shares with adjacent work the idea that quality can be anchored to language, but differs in three respects: it is explicitly video-centric, it keeps a video-aware CLIP-style backbone frozen, and it concentrates adaptation in a unified cross-modal adapter rather than in extensive backbone retraining or purely zero-shot scoring.