Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment

Abstract: Existing vision-LLM (VLM)-based AI-generated image quality assessment (AIGIQA) methods suffer from a fundamental semantic-distortion dimensional conflict: monolithic representations optimized for semantic discrimination inherently entangle compositional understanding with low-level perceptual sensitivity, rendering them blind to fine-grained quality degradations. We introduce MST-CLIPIQA, a multi-scale two-stream framework that achieves hierarchical vision-language alignment through explicit representational decoupling. Our architecture leverages dual CLIP encoders with complementary patch granularities: coarse-grained streams capture global semantic coherence while fine-grained streams preserve textural signatures and artifact patterns. An information bottleneck-inspired gated fusion mechanism performs adaptive cross-scale distillation, with optional cross-attention enabling prompt-anchored correspondence evaluation when generation prompts are available. Extensive experiments across five benchmarks establish new state-of-the-art results, achieving average improvements of 1.11 percent SRCC on quality and 2.35 percent SRCC on text-image correspondence prediction, while maintaining efficiency with only 0.8M trainable parameters. Our project is available at https://github.com/YMlinfeng/MST-CLIPIQA.

• The paper introduces MST-CLIPIQA, a two-stream architecture that decouples global semantic understanding from local distortions for AI-generated images.
• It employs multi-scale feature extraction and gated feature fusion to selectively combine coarse and fine representations, optimizing quality assessment.
• The method achieves state-of-the-art improvements on multiple benchmarks with less than 1M trainable parameters, affirming its efficiency.

Decoupling Semantics from Distortions in AI-Generated Image Quality Assessment

Background and Motivation

The increasing prevalence of AI-generated images (AIGIs) via advanced generative models has created a new set of requirements for perceptual assessment models. Unlike traditional Natural Image Quality Assessment (IQA), AIGIQA must consider not only low-level distortions but also high-level semantics and explicit text-image correspondence, as these attributes directly affect human perception and practical deployment. Existing vision-LLMs (VLMs), particularly CLIP, have demonstrated robust semantic representation, but their monolithic, single-scale design entangles semantic content with perception, leading to a semantic-distortion dimensional conflict: these representations are effective for semantic discrimination but lack sensitivity to subtle artifacts, local degradations, and prompt-image misalignments.

Methodological Advances

The paper proposes MST-CLIPIQA, a Multi-Scale Two-Stream architecture designed explicitly to decouple global semantic understanding from local textural sensitivity. The core idea is that human perceptual judgment is inherently hierarchical, with separate pathways for global composition and local detail analysis. MST-CLIPIQA leverages this by using dual CLIP-based visual encoders with complementary patch granularities—one coarse-grained for semantics, one fine-grained for perceptual distortion.

The architecture comprises three main components:

1. Multi-Scale Two-Stream Feature Extraction (MSTFE): Two parallel streams process the image with distinct patch sizes ($P_c > P_f$). The coarse stream captures the holistic scene and compositional plausibility; the fine stream retains high-frequency texture and detects local artifacts. Both streams use frozen CLIP backbones, maintaining architectural modularity and parameter efficiency.
2. Gated Feature Fusion (GFF): To integrate the complementary streams, the GFF module performs adaptive, per-dimension selection between features. A learnable gating function computes selection coefficients for each embedding dimension, interpolating between coarse and fine representations. This mechanism is formally motivated by the Information Bottleneck principle: the GFF acts as a selective compression function, maximizing mutual information with the quality label while suppressing irrelevant cross-scale redundancies.
3. Prompt-Anchored Cross-Modal Alignment: When generation prompts are available, the model augments fused visual features with explicit cross-attention to the linguistic embedding of the prompt. This provides direct, per-dimension supervision for semantic correspondence, addressing catastrophic prompt-image mismatches that typical VLM embeddings cannot resolve.

The MST-CLIPIQA framework is highly parameter-efficient, with only $\sim$0.8M trainable parameters layered atop frozen visual/textual encoders.

Experimental Results

Evaluation is performed across five contemporary AIGIQA benchmarks, covering both "quality" and "authenticity," as well as explicit text-image alignment prediction tasks. Compared to prior state-of-the-art approaches—including both convolution-based (LinearityIQA, MUSIQ, HyperIQA) and recent transformer- and VLM-based models (StairIQA, MANIQA, LIQE, AMFF-Net, CLIP-AGIQA)—MST-CLIPIQA achieves robust, consistent improvements:

• Quality and Authenticity: Gains of up to $1.11\%$ in SRCC and $0.98\%$ in PLCC over the prior best (LIQE), with pronounced advantages on challenging datasets emphasizing generative distortions and semantic anomalies.
• Text-Image Correspondence: When prompts are provided (MST-CLIPIQA*), the model exceeds previous methods by $2.35\%$ (SRCC), highlighting the benefit of explicit cross-modal reasoning.
• Parameter Efficiency: All results are obtained with less than $1$M trainable parameters, demonstrating favorable computational trade-offs relative to full-network fine-tuning.

Ablation studies validate the necessity of both multi-scale encoding (clear improvement over either scale alone) and gated fusion (outperforming linear and pooling-based integration). The cross-attention mechanism, when enabled, particularly enhances correspondence alignment without degrading other metrics.

Theoretical Justification

MST-CLIPIQA's GFF module is grounded in the Information Bottleneck (IB) formulation. By treating the concatenated multi-scale feature set as the "source" and the regression label as "target," the GFF layer acts as a deterministic mapping bottleneck that compresses redundant representations, preserving only the information relevant for the quality assessment target. This theoretical perspective provides both conceptual clarity and practical guidance for feature selection and gate regularization.

Implications and Future Directions

The MST-CLIPIQA framework makes a substantive contribution to both the practical and theoretical dimensions of AIGIQA:

• Practical Impact: The parameter-efficient, modular design and its plug-in/agnostic compatibility with different backbone encoders enable rapid integration into real-world AIGI evaluation pipelines. Models can be deployed on resource-constrained hardware, with strong generalization even in cases of complex prompt-image relationships and subtle generative artifacts.
• Theoretical Significance: By explicitly decoupling semantics from perceptual distortion, the approach provides a template for hierarchical representation design in VLM-based quality assessment, potentially generalizable to other multi-modal evaluation tasks.
• Future Directions: Extensions could target more aggressive cross-modal grounding, adaptive scale selection, or online learning under dynamic generative model shifts. Incorporating larger VLMs or training the vision encoder end-to-end with the IB principle may yield further improvements. Investigating hard-gated feature routing (instead of soft gating) and developing interpretable gate regularization techniques are promising next steps.

Conclusion

MST-CLIPIQA establishes a new state-of-the-art in AI-generated image quality assessment by systematically decoupling semantic understanding and low-level perceptual sensitivity via a dual-stream, multi-scale architecture. The use of an information bottleneck-inspired gating mechanism and optional cross-modal attention yields strong empirical results and compelling theoretical motivation. This work provides a robust foundation for future explorations in hierarchical, prompt-aware quality assessment for generative models and multi-modal AI evaluation frameworks.

Reference: "Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment" (2606.16799)