---
title: Compositional Consistency Score
url: https://www.emergentmind.com/topics/compositional-consistency-score
type: topic
---

# Compositional Consistency Score

A compositional consistency score is a rigorous, task-matched metric designed to quantify a model’s ability to maintain coherent, structurally sound, and logically consistent outputs under compositional manipulations. Across modalities—vision, language, and simulation-based inference—the term encompasses several related but technically distinct metrics, all addressing whether a system’s performance or internal representations respect the compositional structure of inputs or reasoning chains.

## 1. Formal Definitions Across Modalities

In vision and scene reasoning, the compositional consistency score often refers specifically to the SCS Similarity Index Measure (SCSSIM), which evaluates scene composition structure (SCS) preservation. For visual question answering (VQA) and language tasks, compositional consistency is operationalized as the proportion of question-sets or reasoning chains for which a model’s correctness persists across systematically constructed compositional variants. In simulation-based inference, a compositional score denotes an aggregation of individual score functions, with theoretical guarantees on error accumulation under composition.

**Image Structure/SCSSIM** ([2508.05037]):
$$
\text{SCSSIM}(I, I_0) = \frac{1}{2}\big(M(I, I_0) + M(I_0, I)\big),
$$
with
$$
M(I,I_0)=\exp(-\lambda \bar{k}^2), \qquad \bar{k} = \frac{1}{\tilde N}\sum_{j=1}^{\tilde N} (\ln c_j - \ln c_{0,j}),
$$
where $c_j$'s are normalized cumulative gain curves derived from hierarchical image partitions.

**Compositional Generalization in VQA** ([2412.13636]):
$$
Cons = \frac{1}{|\mathcal{D}_{\text{CCG}}|} \sum_{T\in\mathcal{D}_{\text{CCG}}} \prod_{(Q,A)\in T} \mathrm{Correct}((Q,A))
$$
where $T$ is a triplet across three composition levels and $\mathrm{Correct}((Q,A))$ is a 0/1 indicator of the model’s accuracy.

**Tree-based Consistency for LLMs** ([2506.12376]):
$$
C_d(\mathcal{F}) = \frac{1}{M} \sum_{m=1}^M \frac{1}{|\mathcal{P}_d(\mathcal{T}_m)|} \sum_{P\in\mathcal{P}_d(\mathcal{T}_m)} sim(v_0, v_d)
$$
where $P$ is a $d$-step path in a self-consistency tree $\mathcal{T}_m$, $sim$ a similarity measure, and $C_d(\mathcal{F})$ the aggregated consistency at depth $d$.

**Contrast Set Consistency** ([2103.09591]):
$$
CCS = \frac{1}{N}\sum_{i=1}^N \mathbf{1} \left[ M(x_i)=y_i \wedge \bigwedge_{j=1}^k M(x_i^{(j)})=y_i^{(j)} \right]
$$
expressing the fraction of original-plus-perturbed question sets on which the model answers every instance correctly.

## 2. Methodological Foundations

Compositional consistency scores are typically derived from controlled perturbations of input data or tasks that isolate specific compositional phenomena:

- **Hierarchical Decomposition**: In SCSSIM, images are decomposed via greedy, variance-maximizing splits into a binary tree of cuboid regions, abstracting away from object-level semantics and focusing on spatial partition structure ([2508.05037]).
- **Programmatic Perturbation**: In VQA, compositional contrast sets are generated by minimally perturbing functional programs or scene graphs, resulting in input-output sets where only compositional elements differ ([2103.09591], [2204.07190]).
- **Multi-level Generalization**: GQA-CCG and similar datasets provide triplets or DAGs of questions, explicitly compositional in their logical structure, to assess whether the model’s accuracy propagates through different granularity levels ([2412.13636], [2204.07190]).
- **Reversible Transformation Trees**: ConsistencyChecker constructs self-consistency trees using pairs of inverse operations, measuring whether repeated application and inversion of such operations yields stable function or meaning ([2506.12376]).

## 3. Mathematical Properties and Theoretical Guarantees

Compositional consistency scores exhibit key properties critical for robust benchmarking:

- **Range and Bound**: All variants produce values in $[0,1]$, with perfect compositional generalization yielding $1$.
- **Monotonicity**: In SCSSIM, consistency decays monotonically under compositional distortions (rotation, scaling, cropping), while remaining invariant under non-structural changes (noise, blur). In contrast sets, failure on any perturbed variant zeros out the whole set’s consistency ([2508.05037], [2103.09591]).
- **Tight Upper Bound**: In multilevel compositions, the overall consistency is tightly bounded above by the minimum accuracy over all subcomponents ([2412.13636]).
- **Asymptotic Consistency**: For simulation-based inference, the compositional score’s mean squared error vanishes as the number of observations grows and individual errors decrease, under natural regularity assumptions ([2510.15817]).

## 4. Practical Usage and Experimental Insights

Empirical results highlight the diagnostic power of compositional consistency scores:

| Domain               | Metric/Score             | Key Result                                | Paper          |
|----------------------|-------------------------|--------------------------------------------|---------------|
| Image SCS            | SCSSIM                  | Noisy/blurred images: SCSSIM$\approx$1, Rotated: 0.09, while other metrics (LPIPS, SSIM, CLIP) misorder cases | [2508.05037] |
| VQA (GQA-CCG)        | Consistency ($Cons$)    | MAC+MLO: 34.10% consistency, strong baseline CFR: 49.27%; state-of-the-art VLMs seldom exceed 45% | [2412.13636] |
| Language model MT/code| $C_3(\mathcal{F})$      | GPT-4o-mini: 98.0% (MT), 76.5% (code); Qwen-2.5-32B: 96.4% (MT), 85.1% (code) | [2506.12376] |
| VQA contrast sets     | CCS                     | LXMERT: 83.9% original, 67.2% perturbed, 46.1% full consistency; retraining with perturbations closes the gap | [2103.09591] |

In all cases, compositional consistency scores reveal failures that are not captured by overall accuracy or conventional similarity metrics, particularly on out-of-distribution compositional variations.

## 5. Critical Comparisons and Limitations

Compositional consistency targets a more stringent regime than standard metrics:

- **Robustness to Nuisance Variation**: SCSSIM and tree-based LLM consistency are robust to superficial changes but sensitive to meaningful compositional structure, distinguishing them from pixel-level or embedding-based metrics ([2508.05037], [2506.12376]).
- **Diagnostic Precision**: Right-for-the-Wrong-Reasons and Internal Consistency metrics in video QA explicitly expose spurious reasoning and logical contradictions ([2204.07190]).
- **Coverage Constraints**: Many frameworks rely on fixed compositional templates or strictly paired DAGs/triplets, limiting their extensibility to other forms of composition or unenumerated structure ([2412.13636], [2103.09591]).
- **Human Alignment Limitation**: Some compositional scores operate reference-free (e.g., ConsistencyChecker), while in other domains, consistency may not align perfectly with human preference or intuition, as in the case of CLIP or FID remaining insensitive to structural flips ([2508.05037]).

## 6. Implementation and Best Practices

Effective use of compositional consistency scoring requires careful protocol adherence:

- **Parameter Choices**: For SCSSIM, 64 partitioning cuts and a decay parameter $\lambda=25$ yielded robust discrimination ([2508.05037]).
- **Scalability**: Complexity scales linearly with both data size and structural complexity in most frameworks (SCSSIM, ConsistencyChecker).
- **Implementation Guidance**: Employ integral images for fast sum-of-squared-error computation, maintain priority queues for greedy partitioning, and use automated or dynamic benchmark generation to avoid information leakage ([2508.05037], [2506.12376]).
- **Usage Recommendations**: For models targeting compositional robustness, combinatorial or contrast-set augmentation of training data can directly improve compositional consistency without degrading in-distribution accuracy ([2103.09591]).

## 7. Future Directions and Extensions

Current compositional consistency frameworks invite further refinement and generalization:

- **Extension to Arbitrary Composition Levels**: Moving beyond triplets or shallow DAGs to n-level or tree-based evaluations would improve granularity ([2412.13636], [2506.12376]).
- **Flexible Partial Consistency Metrics**: Introducing soft penalties or weighting by composition depth may better capture nuanced failures and successes.
- **Domain Generalization**: Porting methodology to novel domains—scientific texts, equation parsing, graph reasoning—would expand diagnostic reach.
- **Integration with Human Judgments**: Blending compositional consistency with human-in-the-loop or adversarial assessment could bridge the gap between structural robustness and practical applicability.

Compositional consistency scores fundamentally shift evaluation from surface-level similarity toward principled measurement of systemic, structural robustness, offering indispensable tools for benchmarking, diagnosing, and improving models in vision, language, and probabilistic inference frameworks.

Source: https://www.emergentmind.com/topics/compositional-consistency-score