PerceptEval: A Human-Aligned Evaluation Framework

• PerceptEval is a framework that defines human-aligned evaluation systems using pairwise comparisons and multi-dimensional annotations across various modalities.
• It employs structured protocols like A/B, ABX, MOS, and MUSHRA tests to replace single-score assessments with reproducible, preference-driven metrics.
• Applied in speech synthesis, image quality, text-to-image generation, and video reasoning, PerceptEval offers actionable insights by focusing on semantic, affective, and cross-modal consistency.

PerceptEval denotes a class of perceptual evaluation frameworks that aim to measure what human observers, listeners, or users actually perceive, rather than only signal fidelity, task accuracy, or proxy statistics. Across recent work, the term is applied to automated or semi-automated evaluation systems for speech expressiveness, image quality, text-to-image generation, video understanding, multimodal judge reliability, and modality reliance. A common thread is alignment to human judgments through pairwise preferences, multi-dimensional annotations, perceptual sub-scores, or verifiable grounding constraints, often with explicit attention to semantics, affect, authenticity, and cross-modal consistency (Lin et al., 23 Oct 2025, Ghazanfari et al., 2024, Wang et al., 11 Apr 2025, Park et al., 1 Jun 2026).

1. Scope and definitional usage

In current usage, PerceptEval is not a single benchmark or metric. It is a broader evaluative orientation in which the target of measurement is human perceptual experience. In speech synthesis, this means “how expressive, engaging, and authentic the speech sounds,” rather than only intelligibility or acoustic cleanliness. In image quality assessment, it means judgments that are sensitive to semantically important regions such as faces and text, rather than only local distortions. In multimodal generation, it means jointly assessing perceptual quality, semantic correspondence, and task-specific faithfulness. In multimodal reasoning and judge evaluation, it means preferring perceptually grounded answers over fluent but visually inconsistent narratives (Lin et al., 23 Oct 2025, Chinen et al., 2018, Wang et al., 11 Apr 2025, Park et al., 1 Jun 2026).

Domain	Representative instantiation	Perceptual target
Speech and TTS	DeEAR; multilingual pairwise TTS evaluation	expressiveness, intelligibility, liveliness, authenticity
Image quality	semantic perceptual image metric; PerceptNet; solar Perception Evaluation	semantic relevance, HVS-inspired distance, texture fidelity
Vision-language and T2I	PercepT; LMM4LMM; UniSim-Bench	factual-affective perception, correspondence, human similarity
Video and VLM evaluation	Perception Test; perception-verified self-training; Perception-Judge	memory, physics, semantics, perceptual grounding
Text-to-audio	AudioEval	enjoyment, usefulness, complexity, quality, alignment

This breadth has two consequences. First, PerceptEval is typically multi-dimensional rather than scalar. Second, it is usually human-aligned by construction: either through direct annotations, structured pairwise comparisons, or surrogate models trained to reproduce human preferences (Lin et al., 23 Oct 2025, Wang et al., 16 Oct 2025, Ghazanfari et al., 2024).

2. Core methodological patterns

A recurrent methodological pattern is the replacement of single-score subjective protocols with structured, reproducible, and often pairwise evaluation. ReSEval formalizes this by supporting A/B, ABX, MOS, and MUSHRA-style tests across audio, image, text, and video, with deployment controlled by a shareable configuration file and stimuli assets. The framework is explicitly designed so that a subjective study can be rerun identically from Python or the command line (Morrison et al., 2022).

Another pattern is 2AFC or triplet-based supervision rather than unrestricted rating. The semantic perceptual image metric learns a full-reference score from pairwise judgments over triplets $(o,a,b)$, using fixed VGG-16 features and a linear combination of per-layer $L_1$ differences,

$f(x, y) = \sum_i w_i \|\phi_i(x) - \phi_i(y)\|_1,$

with logistic regression trained so that the metric preserves human preference orderings (Chinen et al., 2018). UniSim-Bench generalizes the same 2AFC logic across image-image similarity, image-text alignment, text-image alignment, and IQA, then uses task-level and overall accuracy to compare general-purpose, specialized, and unified metrics (Ghazanfari et al., 2024).

A third pattern is explicit decomposition of perceptual judgments into sub-dimensions or modalities. The Perceptual Score paper defines modality perception by the performance drop caused by permuting one modality at test time: $$P_{f, x, y}(M_m) = \text{Acc}_{f, x, y}(\mathcal{M}) - \text{Acc}_{f, x, y}(\mathcal{M} \setminus \{M_m\}),$$ where “removal” is implemented by replacing modality $M_m$ with that modality from another test sample. This makes perceptiveness measurable independently of raw task accuracy and reveals whether a nominally multimodal system is actually relying on the visual, audio, or textual stream (Gat et al., 2021).

LLM-mediated psychophysics provides another variant. LLMs have been prompted to produce pairwise similarity judgments across six modalities—pitch, loudness, color, vocal consonants, taste, and timbre—and their judgments are then correlated with human psychophysical data. This work treats LLMs as a lower bound on the perceptual information recoverable from language alone, and uses standardized prompts, repeated sampling, and bootstrapped Pearson correlations as the evaluation protocol (Marjieh et al., 2023).

3. Speech, TTS, and text-to-audio evaluation

In speech-to-speech evaluation, DeEAR is a PerceptEval framework that converts human preference for speech expressiveness into an objective score. It decomposes expressiveness into Emotion, Prosody, and Spontaneity, learns specialized scorers for each dimension, and then fuses them with an XGBoost regressor trained on fewer than 500 annotated clips. The composite score is

$$S_{\text{expr}} = f_{\theta}\big(S_{\text{Emotion}}, S_{\text{Prosody}}, S_{\text{Spontaneity}}\big),$$

with all scores calibrated to a 0–100 scale. DeEAR reports PCC $= 0.91$ and SRCC $= 0.86$ for overall expressiveness, and SRCC $= 0.96$ for system-level ranking across seven S2S models. It is also used for evaluation-driven data curation: selecting $14\text{K}$ utterances to form the 51-hour ExpressiveSpeech dataset and improving an S2S model’s expressiveness score from 2.0 to 23.4 (Lin et al., 23 Oct 2025).

A related but distinct PerceptEval design appears in multilingual TTS for Indian languages. There, perceptual evaluation is explicitly pairwise and multidimensional, using 5,357 sentences across 10 Indic languages and approximately 120,000 A/B comparisons from 1,915 native raters. Annotation proceeds in two phases: a locked overall preference decision, followed by six perceptual dimensions—intelligibility, expressiveness, voice quality, liveliness, hallucinations, and presence of noise. Ranking is then modeled with Bradley–Terry: $L_1$0 SHAP analysis identifies Expressiveness and Intelligibility as the strongest predictors of overall preference, and the reliability analysis shows that around 100–200 raters and around 1,000 sentences suffice for stable ranks at 5–7 systems (Anand et al., 23 Apr 2026).

For text-to-audio generation, AudioEval extends the same logic to a dual-perspective, multi-dimensional setting. Its dataset contains 4,200 audio clips from 24 systems with 126,000 ratings across Content Enjoyment, Content Usefulness, Production Complexity, Production Quality, and Textual Alignment, separately for experts and non-experts. Qwen-DisQA, built on Qwen2.5-Omni, predicts a full 10-point distribution for each of the five dimensions and both perspectives, and is trained with a KL-plus-MSE objective over smoothed human rating distributions. On utterance-level Pearson correlation, it outperforms zero-shot and fine-tuned baselines across nearly all dimensions, for example reaching expert correlations of 0.725 on Content Enjoyment and 0.726 on Production Quality, and non-expert correlations of 0.671 and 0.738 on the same dimensions (Wang et al., 16 Oct 2025).

Across these audio settings, PerceptEval consistently moves away from monolithic MOS. It instead models perception as multi-attribute, preference-driven, and often listener-specific (Lin et al., 23 Oct 2025, Anand et al., 23 Apr 2026, Wang et al., 16 Oct 2025).

4. Image quality, semantic similarity, and multimodal image perception

In image quality assessment, one of the earliest PerceptEval-style developments is the semantic perceptual image metric built from fixed VGG-16 features. The metric learns only 10 scalar weights—one per selected VGG layer—yet it shows strong responses to semantically important regions such as faces and text, which better matches minimally instructed human raters. The supporting dataset contains 140k unique reference images and about 700k 2AFC judgments. On TID2013, the metric reaches SROCC $L_1$1 and KROCC $L_1$2, ranking in the top 5 of 16 methods, while its heatmaps demonstrate semantic sensitivity absent from more classical contrast-oriented metrics (Chinen et al., 2018).

PerceptNet represents a different line of work: an HVS-inspired perceptual distance model whose architecture mirrors retinal, LGN, and V1 processing. Its perceptual distance is

$L_1$3

Despite having only 36.3k parameters, it achieves strong performance on traditional IQA datasets, including TID2008 Test PLCC/SRCC of 0.93/0.93 and LIVE PLCC/SRCC of 0.95/0.98 when trained on TID2008. The paper also shows that replacing ReLU with GDN in AlexNet improves perceptual similarity modeling on classic IQA benchmarks (Hepburn et al., 2019).

A reduced-reference variant appears in solar imaging. There, “Perception Evaluation” is defined as the cosine similarity between Gram matrices of VGG16 feature maps extracted from a high-resolution reference image and a test image: $L_1$4 This metric is motivated by the multifractal texture structure of solar images, is robust to reference-patch choice and rotation, and decreases monotonically with increasing blur under simulated turbulence (Huang et al., 2019).

PerceptEval in image generation and vision-language modeling broadens from quality to experience and correspondence. LMM4LMM introduces EvalMi-50K, with 50,400 images from 24 T2I models, 100,800 MOS labels across perceptual quality and text-image correspondence, and 50,400 QA labels over 20 task dimensions. LMM4LMM reports overall SRCC $L_1$5 for perception and SRCC $L_1$6 for correspondence, while model-rank alignment to human reaches 0.979 for perception and 0.983 for correspondence (Wang et al., 11 Apr 2025). PercepT further shifts the target from quality to “perception topics,” combining objective factual and subjective affective aspects in image-caption pairs. On ArtELingo, it achieves silhouette score 0.97 versus 0.37 for the closest baseline and mapping AUC 0.94 versus 0.77 (Mohamed et al., 2 Jun 2026).

These systems collectively indicate that image-centric PerceptEval has moved from distortion sensitivity toward semantic salience, affective interpretation, and prompt-conditioned perceptual alignment (Chinen et al., 2018, Wang et al., 11 Apr 2025, Mohamed et al., 2 Jun 2026).

5. Unified multimodal benchmarks, video diagnostics, and judge grounding

A major development is the attempt to unify perceptual evaluation across modalities and task structures. UniSim-Bench organizes 7 tasks and 25 datasets into a shared 2AFC framework spanning image-image similarity, image-text alignment, text-image alignment, IQA, perceptual attribute assessment, odd-one-out, and image retrieval. The main result is double-edged: specialized metrics excel on their native tasks, and unified multi-task fine-tuning yields the best average performance on core tasks, but both specialized and unified models still struggle on far-OOD tasks, especially odd-one-out and retrieval (Ghazanfari et al., 2024).

Perception Test extends the same diagnostic ethos to video. It contains 11,609 real-world videos with audio, average length about 23 seconds, and six annotation types: multiple-choice QA, grounded video QA, object tracks, point tracks, temporal action segments, and sound segments. The benchmark is organized around four skills—Memory, Abstraction, Physics, Semantics—and four reasoning types—descriptive, explanatory, predictive, counterfactual. The human baseline on MC-QA is 91.4%, while the strongest reported model result in the summary is 46.2% zero-shot, exposing a large gap between current multimodal models and human perceptual reasoning (Pătrăucean et al., 2023).

PerceptEval also appears as an internal verifier inside training pipelines. In perception-verified self-training for VLMs, the model is forced to emit a caption–reasoning–conclusion template, and the caption is screened by a PerceptEval module that jointly checks OCR agreement and visual agreement. The decision rule is

$L_1$7

with thresholds derived from the OCR text-area ratio. Only captions that satisfy both checks are admitted to the “easy” or “medium” self-training subsets. Ablations show that dual filtering improves substantially over answer-only filtering, and combining OCR agreement with FG-CLIP yields the best results (Sharma et al., 20 Jun 2026).

A related but distinct use arises in multimodal judges. The Perceptual Judgment Bias study shows that MLLM judges frequently reward plausible but perceptually wrong responses when visual evidence conflicts with text. It constructs the Perceptually Perturbed Judgment Dataset with target ordering $L_1$8, then trains judges with a GRPO-based batch-ranking reward: $L_1$9 Perception-Judge-Flex-7B reaches 85.7% accuracy in the bias decomposition benchmark, versus 69.5% for Qwen2.5-VL-7B and 76.6% for Flex-Judge-VL-7B, with reductions in both insufficient-perception errors and response-anchored judgment errors (Park et al., 1 Jun 2026).

6. Reproducibility, limitations, and recurring controversies

A central claim across this literature is that human evaluation remains necessary but is too slow, costly, or unstable to serve as the only standard. MOS is described as slow, expensive, and inconsistent in speech; expert-rater image protocols may diverge from minimally instructed user judgments; and video reasoning benchmarks remain expensive to annotate densely (Lin et al., 23 Oct 2025, Chinen et al., 2018, Pătrăucean et al., 2023). PerceptEval therefore tends to replace direct subjective testing with calibrated surrogates, but this substitution is always partial.

Several recurring limitations are explicit. Domain and language coverage remain narrow in many systems: DeEAR is strongest for English and Chinese conversational speech; PercepT relies on translation of non-English captions into English; AudioEval does not report Cronbach’s alpha or ICC; and solar reduced-reference evaluation requires same-wavelength reference imagery (Lin et al., 23 Oct 2025, Mohamed et al., 2 Jun 2026, Wang et al., 16 Oct 2025, Huang et al., 2019). Generalization also remains unresolved. UniSim shows that unified metrics improve average core-task performance but still underperform on structurally different OOD tasks, and Perception-Judge still leaves room for improvement on extremely fine-grained perceptual cues and reasoning-heavy tasks (Ghazanfari et al., 2024, Park et al., 1 Jun 2026).

One persistent misconception is that higher benchmark accuracy implies stronger perceptual grounding. The Perceptual Score work shows the opposite can happen: more accurate multimodal models may “perceive” the visual modality less than their predecessors, because they exploit textual priors or dataset shortcuts more effectively (Gat et al., 2021). A related controversy concerns semantic weighting. In consumer-facing imaging, emphasis on faces, text, or salient objects may better reflect ordinary users’ judgments; in scientific or medical settings, that same weighting may be undesirable because strict low-level fidelity matters יותר. The semantic image metric paper states this contrast explicitly for real-world applications versus domains such as medical or scientific imaging (Chinen et al., 2018).

The literature also converges on several future directions. These include using human-aligned perceptual metrics as rewards for RLHF-style optimization in speech synthesis, extending perceptual evaluation to more languages and cultures, adding region-level grounding and multi-scale semantic modeling, and standardizing judge evaluation under controlled perceptual perturbations (Lin et al., 23 Oct 2025, Mohamed et al., 2 Jun 2026, Park et al., 1 Jun 2026). Collectively, these works suggest that PerceptEval is evolving from a collection of task-specific metrics into a more general research program: the systematic objectification of human perceptual judgment across modalities, with explicit attention to semantics, affect, grounding, and reproducibility.