--- title: Mean Opinion Scores (MOSs) url: https://www.emergentmind.com/topics/mean-opinion-scores-moss type: topic --- # Mean Opinion Scores (MOSs) Mean Opinion Scores (MOSs) are scalar summary measures derived from subjective human evaluations, widely used to assess the perceived quality of generated content in multimedia applications, especially in image- and video-based virtual try-on systems. MOSs serve as key ground-truth references for benchmarking computational image synthesis methods, including those employing GANs and diffusion models, by condensing large volumes of human ratings into a concise quantitative metric. ## 1. Definition and Mathematical Basis A Mean Opinion Score is classically defined as the arithmetic mean of a set of human subject ratings $S_1, S_2, \ldots, S_N$ over a fixed scale (typically 1–5 or 1–10, where higher scores denote higher perceived quality or realism): \[ \mathrm{MOS} = \frac{1}{N} \sum_{i=1}^N S_i \] Here, each $S_i$ is a subjective judgment, for example on a Likert scale or as a forced-choice preference mapped to numerical values. Statistical analysis assumes all $S_i$ are sampled IID from the underlying subjective distribution per sample. ## 2. MOS in Virtual Try-On: Roles and Methodological Protocols In image-based virtual try-on (VTON), MOSs are deployed as primary indicators of system-level perceptual quality—encompassing naturalness, faithfulness, garment-body alignment, and plausibility under varied poses. State-of-the-art works in vision-based try-on, such as CatV2TON [2501.11325], report MOSs to compare against both previous GAN-based and diffusion-based pipelines. Protocols routinely involve: - Selection of generated try-on images (and, optionally, ground truth or baseline outputs). - Recruitment of human raters—either trained experts or crowdsourced workers (e.g., via Amazon Mechanical Turk). - Standardized presentation instructions: "Please rate the photo-realism and garment realism of this person’s outfit." - Use of a closed set of ratings (e.g., 1–5, integers). - Replicates and randomization to minimize presentation and order bias. For robust estimation, contemporary studies prefer >30 raters per sample and randomized presentation. Outlier rejection or z-score normalization is sometimes used to control for rater calibration drift. ## 3. Statistical Properties and Interpretation MOSs exhibit statistical properties governed by the sampling scheme: - The sample MOS is an unbiased estimate of the population mean subjective rating for the stimulus. - Confidence intervals can be derived via the central limit theorem as \[ [\mathrm{MOS} - z \, \sigma / \sqrt{N},\; \mathrm{MOS} + z \, \sigma / \sqrt{N}] \] where $\sigma^2$ is the sample variance and $z$ is the quantile from the normal distribution. A key consideration is inter-rater variability. While the mean gives a global quality metric, best practice also reports standard deviation or median to reflect subjective dispersion, especially in datasets showing multimodality in perception (e.g., due to artifacts or diverse cultural standards). ## 4. MOS versus Computational Perceptual Metrics in Research Practice Although image-based try-on systems are evaluated by paired and unpaired computational metrics (e.g., SSIM, FID, LPIPS), MOSs remain the definitive external perceptual reference [2501.11325, 2303.13756, 2511.15831]. For example, CatV2TON [2501.11325] is benchmarked using SSIM, LPIPS, FID, KID as well as human opinion scores in comprehensive experimental validation. GP-VTON specifically incorporates Human Evaluation (HE, an MOS measure), with human raters indicating perceptual realism percentages, showing high MOS favorability over baselines (e.g., GP-VTON achieves 50.9% MOS-based HE on VITON-HD). Notably, the literature establishes that high MOS alignment with low FID/LPIPS and high SSIM is not guaranteed. MOS is sensitive to subtle aspects (garment-body misalignment, semantic inconsistencies, photo-compositing artifacts, user preference for specific textures) that may escape reference-free or reference-based computational metrics [2501.11325, 2303.13756, 2311.04811]. ## 5. Advanced and Variant MOS Protocols Several VTON studies introduce MOS variants to dissect specific qualities: - Pairwise Preference MOS: Each participant rates, for a set of systems A/B, which output appears more realistic or faithful. The MOS is then computed as the percentage of wins per system, e.g., “GP-VTON (ours) achieves 50.9% MOS-based human preference.” - Attribute-specific MOS: Raters score realism separately for face, cloth alignment, garment texture, or scenario-specific criteria (e.g., sleeve-hand alignment). PG-VTON and PL-VTON [2503.14074, 2503.12588] report MOS for limb region realism. - Task-specific MOS: In multi-task VTON (single-garment, model-to-model, multi-garment), UniFit [2511.15831] computes MOS per task and averages across synthetic scenarios, enabling fine-grained performance assessment. ## 6. Best Practices and Limitations Well-designed MOS acquisition requires: - Large, demographically diverse rater pools. - Anonymization and randomization of reference/baseline identities. - Clear guidance and quality control (e.g., attention checks or consistency questions). Limitations are acknowledged: - Training and rater fatigue can bias MOS downward in long surveys. - MOS does not necessarily reflect target population preferences if raters differ systematically from end users. - Subtle differences in instruction framing can yield significant MOS variation across studies. A plausible implication is that the reported MOSs in the VTON literature provide a robust and reproducible, though not entirely absolute, measure of end-user experience. While MOSs remain the gold-standard perceptual metric, they should be interpreted alongside auxiliary computational metrics for comprehensive system validation. ## 7. MOS Reporting in the State-of-the-Art VTON Literature The following table summarizes the reporting of MOSs and human evaluation measures in representative VTON works: | Framework | MOS/HE Protocol | Key Quantitative Human Results | |----------------|------------------------|---------------------------------------| | CatV2TON [2501.11325] | MOS (photorealism, garment alignment) | Best SSIM, 2nd-best FID/KID, high subjective realism | | GP-VTON [2303.13756] | Human Evaluation (HE, % MOS-like) | 50.9% on VITON-HD, superior to all baselines | | UniFit [2511.15831] | MOS per complex scenario, multi-garment | Consistently leading or matching SOTA in human rating | | C-VTON [2212.04437] | MTurk pairwise preference (MOS) | 52–76% preference over baselines | | PL-VTON [2503.12588/2503.14074] | User preference/MOS for sleeve/limb handling | 62–83% in A/B studies, >75% realism for limbs | | PG-VTON [2304.08956] | User study (perceptual fit/skin realism) | Outperforms all baselines in high-difficulty scenarios | In sum, Mean Opinion Scores provide a statistically grounded, human-centered, and broadly adopted standard for benchmarking perceptual quality in VTON research. Their rigorous deployment is critical for objectively comparing increasingly sophisticated image- and video-based virtual try-on models. Source: https://www.emergentmind.com/topics/mean-opinion-scores-moss