Consistency Quality Score (CQS)

• CQS is a quantitative metric family that measures the stability, reliability, and semantic coherence of outputs in machine learning across domains.
• It employs rigorous computation using pairwise similarity, correctness indicators, and statistical aggregation to assess repeatability and model consistency.
• CQS guides both diagnostic evaluations and comparative benchmarking by revealing reporting discrepancies and informing model selection.

The Consistency Quality Score (CQS) is a family of formal, quantitative metrics designed to measure the stability, reliability, and semantic coherence of outputs or evaluations in machine learning and related disciplines. Across domains such as image generation, video synthesis, classification, natural language processing, and model-vs-model evaluation, CQS variants operationalize the notion of "consistency" through rigorous computation rooted in statistical or semantic alignment, repeated trials, or cross-metric agreement. Its use spans both diagnostics (e.g., detecting reporting errors or output variance) and comparative benchmarking (e.g., model ranking, test robustness, corpus health monitoring).

1. Mathematical Definitions and Key Formulations

CQS metrics are instantiated according to domain and evaluation purpose. Below, representative formulations are presented:

@@@@1@@@@ (Image Generation):

Given $N$ repeat generations $\{I_1,\ldots,I_N\}$ from a fixed prompt, embed each image via (e.g.) CLIP ViT-B/32 to yield normalized vectors $E_i\in\mathbb{R}^{512}$. Define the semantic CQS as

$$\mathrm{CQS} = \frac{2}{N(N-1)}\sum_{1\leq i < j \leq N} \max(100 \cdot \cos(E_i, E_j), 0)$$

This gives a [0,100] score, with higher values indicating greater semantic repeatability (Bent, 2024).

Sample Learning Consistency (Curriculum Learning):

For input $x_i$, $M$ independently initialized models, and $T$ epochs, CQS is the average indicator of correctness:

$$c_i = \frac{1}{MT} \sum_{m=1}^M \sum_{t=1}^T I_{m, t}(i), \quad I_{m,t}(i) = \text{correct in run } m \text{ at iteration } t$$

Ranges in $[0, 1]$; high CQS signals examples consistently learned across runs (Raymond-Saez et al., 2022).

Text Data (n-gram Consistency):

Given an $n$-gram LLM forecast $\widehat{w}_i$ for each token $w_i$ in a sequence of $N$ tokens:

$$\mathrm{CQS} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}\{\widehat{w}_i = w_i\}$$

Internal CQS refers to self-consistency (model fit on same text); external CQS uses an external corpus (Chiu et al., 2021).

LLM Matchup Judging:

For a judge model over matchups $M$ between models $(i, j)$, with win probability $p_{ij}$ and $n_{ij}$ trials per pair:

$$\overline{\mathrm{Var}} = \frac{\sum_{(i,j)\in M} n_{ij}p_{ij}(1-p_{ij})}{\sum_{(i,j)\in M} n_{ij}},\quad \mathrm{CQS} = 1 - 4 \overline{\mathrm{Var}}$$

A value near $1$ signifies deterministic, high-resolution judging; near $0$ indicates random guessing (Ramaswamy et al., 27 Sep 2025).

Multiple Choice Robustness (Consistency-Rebalanced Accuracy):

Given $M$ variants of each of $N$ items, with model correctness over all variants:

• Per-item repeated-correctness $RC(i) = (1/M)\sum_{c=1}^M \mathrm{LLM}(mcq^c_i)$
• Bare-Minimum-Consistency Accuracy:

$$\mathrm{BMCA}(1.0) = \frac{1}{N}\sum_{i=1}^N \mathbb{I}\{RC(i)=1\}$$

• Consistency Index: $CI = 1.0 - (MCQA - BMCA(1.0))$
• Rescaled Accuracy: $CoRA = MCQA \cdot CI$ (Cavalin et al., 26 Nov 2025)

Contrast-Set Relative Consistency:

Given accuracy $a$ and observed bundle-consistency $c$ across $n$ bundles, relative CQS is:

$$RC(c, a) = \sum_{i=C_{min}(a)}^{c} \frac{m(i, a)}{M(a)}$$

where $m(i, a)$ counts allocations of $a$ correct items yielding $i$ fully-correct bundles; $M(a)$ is the total $\binom{2n}{a}$.

Binary Classifier Report Consistency:

Given reported metrics (accuracy, sensitivity, specificity) and test set $(p, n)$, over all valid confusion matrices $(tp, tn)$, total violation is

$$\Delta^* = \min \left\{ \sum_{s=1}^M \max[0, f_s(tp, tn) - (\widehat{v}_s+\epsilon_s), (\widehat{v}_s-\epsilon_s)-f_s(tp, tn)] \right\}$$

Normalized CQS:

$$\mathrm{CQS} = 1 - \frac{\Delta^*}{\sum_{s=1}^M \epsilon_s}$$

(Fazekas et al., 2023)

Video Generation (World Consistency Score):

CQS is a learned non-negative linear combination of:

• $M_{perm}$: Object permanence (disappearance/appearance is penalized)
• $M_{rel}$: Relation stability (spatial relations and smoothness)
• $M_{causal}$: Causal compliance (motion explained by plausible events)
• $M_{flicker}$: Frame-wise flicker (based on optical flow) Weights are fitted via ridge regression to maximize alignment with human MOS on standard video benchmarks (Rakheja et al., 31 Jul 2025).

2. Computation and Implementation

Across use cases, CQS computation typically consists of the following steps:

1. Raw Data Collection: Multiple outputs or judgments per input, e.g., repeated image generations, distinct data folds, or shuffled MC variants.
2. Feature Extraction or Metric Calculation: CLIP embeddings for images; correctness indicators for classification; token matches for text; object tracks and flows for video.
3. Pairwise or Aggregate Scoring: Calculation of pairwise similarities (e.g., cosine), correctness averages, or metric feasibility under rounding.
4. Statistical or Numerical Aggregation: Averaging, maximization, or probabilistic aggregation depending on target metric.
5. Normalization: To a fixed range, typically $[0, 1]$ or $[0, 100]$.

Domain-specific pseudocode is provided in original works; e.g., the image-generation CQS pseudocode (Section 2c in (Bent, 2024)) and confusion-matrix search routine for classifier-report CQS (Fazekas et al., 2023).

Implementation frequently leverages open-source libraries: aRianna for text CQS (Chiu et al., 2021), mlscorecheck for reporting CQS in Python (Fazekas et al., 2023), and standard evaluation toolkits, object trackers, or LLM APIs for domain-specific tasks.

3. Empirical Properties, Benchmark Results, and Human Alignment

CQS metrics have been validated against human or practical standards in multiple domains:

• Diffusion models: Semantic CQS matched human annotator majority-choices in 94% of prompts (per-prompt side-by-side gallery comparison) (Bent, 2024).
• LLM Elo proxy: Model CQS correlated at $r = 0.91$ with human-derived Elo scores over 24 judge LLMs. Regression from CQS to Elo ranked models with mean absolute error of 35.2 Elo points (Ramaswamy et al., 27 Sep 2025).
• Multiple-Choice Consistency: CoRA sharply penalizes models with high vanilla accuracy but poor consistency across distractor-altered variants (e.g., MedLlama3 dropped from $0.74$ MCQA to $0.32$ CoRA due to low variant-consistency) (Cavalin et al., 26 Nov 2025).
• Video Consistency: Learned CQS achieved Pearson correlation of $0.80$ (VBench-2.0) and $0.78$ (EvalCrafter) versus human mean opinion scores, outperforming classical metrics like FVD (Rakheja et al., 31 Jul 2025).
• Binary Classifier Reporting: CQS precisely detects (in)consistency down to rounding limits, with zero false positives/negatives. Actual medical applications demonstrated robust detection of inconsistent or erroneous reporting (Fazekas et al., 2023).
• Curriculum Learning/Data Difficulty: Individual C-Score (CQS) poorly predictable from only pixels; performance correlates weakly in- and fails out-of-distribution, indicating global-data context is essential (Raymond-Saez et al., 2022).

4. Interpretation, Best Practices, and Limitations

CQS provides interpretable indicators:

• Absolute values index the degree of semantic or statistical repetition: high for low intra-run variability or perfect report-metric alignment; low where stochasticity, mode collapse, or reporting inconsistencies arise (Bent, 2024, Fazekas et al., 2023).
• Relative differences (e.g., 3–5 points in diffusion model SCS) are statistically significant and discriminative between models (Bent, 2024).
• For model selection, high CQS is optimal for production/automation tasks, while lower CQS may indicate creative diversity where that is preferred (Bent, 2024, Cavalin et al., 26 Nov 2025).
• In reporting, CQS exposes infeasibility or misreporting with deterministic guarantees, especially relevant where confusion-matrix counts are integral (Fazekas et al., 2023).

Best practices across applications include:

• Sufficient replicates or variants (e.g., $N=20$ for image prompts, $M=20$ MCQ variants) for stable estimates.
• Fixed random seeds, standardized hyperparameters, and transparent reporting of computation settings (Bent, 2024, Cavalin et al., 26 Nov 2025).
• Use of domain-appropriate embeddings (CLIP for vision, LLMs for text).
• Joint reporting of CQS and classical accuracy or diversity metrics for full model characterization.

Known limitations include:

• Dependency on embedding model biases (e.g., cultural, gender in CLIP) (Bent, 2024).
• Difficulty in cross-domain or highly abstract prompt evaluation.
• Computational intensity for some definitions (sample difficulty CQS requires several full training replicates) (Raymond-Saez et al., 2022).
• Inability to fully resolve among top-tier LLMs (CQS saturates within high-Elo models) (Ramaswamy et al., 27 Sep 2025).
• For MC benchmarks, tailoring variant-generation to avoid artificially hard/easy distractors is required to ensure fair CI (Cavalin et al., 26 Nov 2025).

5. Variants and Extensions Across Domains

CQS conceptualizations are unified under the theme of repeatability or semantic alignment, but operationalized differently:

Domain	Core Principle	Typical Output Range
Image Generation	Pairwise embedding sim.	0, 100
Video	Weighted submetrics	0, 1
Classification	Metric feasibility	[0, 1]
LLM Judging	Aggregated variance	[0, 1]
MCQA Robustness	Consistency-index scaling	[0, 1]
Text Consistency	Proportion token match	[0, 1]
Contrast Sets	Relative consistency (RC)	[0, 1]

In addition, CQS-inspired metrics have been adapted for video artifact detection (object permanence, relation stability, causality, flicker), contrast-vs-accuracy model evaluation (relative consistency), and performance reporting fidelity.

Recommended directions for future extension include:

• Cross-modal CQS (text, audio, video)
• Style/attribute-conditioned consistency measures
• Integration with diversity and creativity metrics to map the consistency-diversity spectrum
• Hybrid local-global CQS for adaptive curriculum learning

6. Theoretical and Practical Impact

CQS has redefined the evaluation landscape in several contexts:

• Model Comparison: Outperforms simple accuracy, FID, Frechet video distances, and naive Elo in alignment with human reliability scales.
• Benchmark Robustness: Discovers hidden brittleness or reporting artifacts missed by aggregate scores.
• Interpretability: Facilitates identification of model failure modes via sub-metric deconstruction (e.g., flicker vs. causal pathology in video).
• Open Science Tools: Multiple CQS variants have associated open-source packages—e.g., aRianna for text (Chiu et al., 2021), mlscorecheck for binary metrics (Fazekas et al., 2023).

CQS development reflects a broader recognition that reliability, trustworthiness, and semantic alignment—rather than sheer accuracy—are indispensable for the robust deployment and fair comparison of AI models. Its adoption continues across computer vision, NLP, benchmarking, and meta-evaluation research (Bent, 2024, Ramaswamy et al., 27 Sep 2025, Cavalin et al., 26 Nov 2025, Johnson et al., 2023, Rakheja et al., 31 Jul 2025, Fazekas et al., 2023, Raymond-Saez et al., 2022, Chiu et al., 2021).