VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression (2512.15701v1)

Abstract: Evaluations of image compression performance which include human preferences have generally found that naive distortion functions such as MSE are insufficiently aligned to human perception. In order to align compression models to human perception, prior work has employed differentiable perceptual losses consisting of neural networks calibrated on large-scale datasets of human psycho-visual judgments. We show that, surprisingly, state-of-the-art vision-LLMs (VLMs) can replicate binary human two-alternative forced choice (2AFC) judgments zero-shot when asked to reason about the differences between pairs of images. Motivated to exploit the powerful zero-shot visual reasoning capabilities of VLMs, we propose Vision-LLMs for Image Compression (VLIC), a diffusion-based image compression system designed to be post-trained with binary VLM judgments. VLIC leverages existing techniques for diffusion model post-training with preferences, rather than distilling the VLM judgments into a separate perceptual loss network. We show that calibrating this system on VLM judgments produces competitive or state-of-the-art performance on human-aligned visual compression depending on the dataset, according to perceptual metrics and large-scale user studies. We additionally conduct an extensive analysis of the VLM-based reward design and training procedure and share important insights. More visuals are available at https://kylesargent.github.io/vlic

• The paper demonstrates that state-of-the-art vision-language models, without fine-tuning, can mimic human binary 2AFC judgments on image quality.
• The paper introduces VLIC, a diffusion-based framework that uses Direct Preference Optimization with VLM ratings to enhance perceptual quality over traditional metrics.
• The paper validates its approach through extensive experiments on datasets like MS-COCO and CLIC, showing improved human alignment and reliable perceptual outcomes.

Vision-LLMs as Perceptual Judges for Human-Aligned Image Compression

Introduction and Motivation

The paper "VLIC: Vision-LLMs As Perceptual Judges for Human-Aligned Image Compression" (2512.15701) critically examines the limitations of traditional image compression evaluation metrics and introduces a new framework that leverages Vision-LLMs (VLMs) as automatic perceptual judges. Conventional metrics such as MSE, PSNR, and SSIM have been established as poorly correlated with human perception, often contradicting subjective assessments of visual quality. Existing approaches typically employ learned perceptual metrics, trained on large-scale human-annotated datasets, but these models exhibit limited generalization and are susceptible to null-space exploitation, resulting in suboptimal perceptual alignment.

This work presents a strong empirical claim: state-of-the-art VLMs, notably Gemini 2.5-Flash, can replicate human binary two-alternative forced choice (2AFC) judgments on standard benchmarks in a zero-shot setting. The authors leverage the visual reasoning capabilities of VLMs without explicit fine-tuning, proposing VLIC—a diffusion-based image compression system calibrated directly via binary VLM ratings, rather than distilling those preferences into a static perceptual metric. The central thesis is that VLM-driven post-training, operationalized through Direct Preference Optimization (DPO) adapted to diffusion autoencoders, leads to compression systems whose visual reconstructions are better aligned with human perception, as demonstrated through both large-scale user studies and perceptually relevant metrics.

Methodology

VLIC integrates a diffusion autoencoder architecture, discrete quantization via finite scalar quantization (FSQ), and a secondary autoregressive entropy coder. The core innovation lies in the training approach: after an initial phase following FlowMo discretization, the model undergoes DPO post-training, using VLM-judged pairwise preferences to drive optimization. The process is illustrated as follows: the system encodes an image into a 1D discrete latent sequence, entropy codes via a LLM, and then decodes stochastically using a conditional diffusion decoder. For each latent code, two reconstructions are sampled and compared. The VLM receives the original and both reconstructions, returning a scalar score reflecting which reconstruction is perceptually closer to the source. These binary judgments determine the winner and loser for the DPO update, allowing for non-differentiable perceptual supervision.

Figure 1: VLIC method overview—encoding, entropy coding, stochastic diffusion decoding, VLM-based ranking, and post-training via Diffusion DPO.

Critical to the reliability of the VLM reward signal are several mitigation strategies: the system ensembles VLM predictions across multiple random seeds; it reverses image order to detect and filter out inconsistent rankings; and crucially, it further ensembles VLM ratings with traditional LPIPS scores, only accepting training pairs when the two metrics concur. This hybridization ensures consistency and reduces the noise characteristic of VLMs when two reconstructions are highly similar.

Experimental Evaluation

Quantitative and Qualitative Analysis

VLIC is comprehensively benchmarked against HiFiC, PerCo, HFD, and PO-ELIC across MS-COCO, CLIC 2020, and CLIC 2022 datasets. The experiments demonstrate that VLIC establishes competitive or state-of-the-art results depending on the dataset and evaluation metric. Notably, VLIC achieves superior performance on perceptual metrics (ELO, FD-DINO), especially on MS-COCO, which contains rich human-relevant features, such as text and faces. Human ELO ratings and large-scale user studies corroborate the alignment of VLIC reconstructions with human judgments.

Figure 2: Qualitative comparison of VLIC against HiFiC and PO-ELIC on CLIC 2022 and MS-COCO, showing superior preservation of perceptually salient details.

Figure 3: Quantitative evaluation on MS-COCO, CLIC 2020, and CLIC 2022 showing VLIC's strong performance on perceptually correlated metrics and human ELO.

Ablation studies reveal the importance of each noise mitigation strategy: self-ensembling of VLM scores significantly improves predictive alignment with human ratings, as higher numbers of seeds reduce variance and increase stability.

Figure 4: Increasing the number of VLM ensemble seeds enhances prediction accuracy on human judgment datasets.

Consistent with the system's design, VLIC tends to prioritize improvement on human-correlated metrics at the expense of pixel-aligned metrics (PSNR), in accordance with rate-distortion-perception theory.

Replication of Human Judgments

The authors demonstrate that Gemini 2.5-Flash matches human accuracy on binary 2AFC tasks. On the BAPPS-val dataset and a custom set of compressed images, Gemini's zero-shot accuracy approaches that of human annotators, outperforming alternative learned metrics in some settings.

Failure Modes and Limitations

Despite overall strong performance, inherent VLM noise remains a challenge, especially for very similar reconstructions. The paper illustrates how inconsistent rankings can arise, and how ensembling partially mitigates these issues but does not eliminate them.

Figure 5: Example of a failure mode—VLM produces self-inconsistent rankings when presented with highly similar reconstructions in reverse order.

The authors also explore edge cases induced by alternative reward formulations. For example, when using an edit-distance reward on readable text, the system degenerates to censoring all readable text in the reconstructions.

Figure 6: Failure case—VLM edit-distance reward incentivizes reconstruction to censor all readable text, illustrating pitfalls of reward specification.

Generalization and Tiled Inference

VLIC supports arbitrary image resolutions at test time via a tiled inference strategy and overlapping border margins for joint diffusion, mitigating artifacts at patch boundaries. This approach preserves visual quality for high-resolution applications.

Figure 7: Tiled inference for arbitrary resolutions, showing original, tessellation strategy, and reconstructed image, with appropriate margin selection to avoid artifacts.

Implications and Future Directions

The findings suggest VLM-based perceptual judges represent a scalable path for human-aligned evaluation and supervision of generative models. As VLM architectures continue to improve, their zero-shot perceptual priors could further elevate the alignment of compression systems to subjective human preferences, potentially rendering static perceptual metrics obsolete. However, reliance on VLMs introduces robustness concerns due to their vulnerability to order, adversarial perturbations, and latent hallucination. Future research will need to address these limitations, refine ensemble strategies, and examine methods for diagnostic validation of VLM-based rewards. The approach may generalize beyond image compression to other generative and conditional modeling domains where human-centric assessment is essential, including video, 3D, and cross-modal synthesis.

Conclusion

VLIC establishes that state-of-the-art VLMs can operate as automatic perceptual judges, driving human-aligned diffusion-based image compression via DPO. The method achieves competitive or superior performance to existing baselines—particularly on metrics correlated with human perception—and introduces a practical framework for integrating VLM-based supervision in post-training regimes. The work underscores the scalability and efficiency advantages of using VLMs for perceptual alignment and points to ongoing challenges in reward specification and reliability. As VLMs evolve, their incorporation into compression and generative modeling pipelines will likely expand, catalyzing advances in subjective quality and evaluative automation.