--- title: Decomposed GPT Score (D-GPTScore) url: https://www.emergentmind.com/topics/decomposed-gpt-score-d-gptscore type: topic --- # Decomposed GPT Score (D-GPTScore) Decomposed GPT Score (D-GPTScore) denotes a decomposition-based scoring paradigm in which a large language model or multimodal large language model evaluates an output by splitting the judgment into finer-grained sub-assessments and then aggregating them into a final score. In the concept customization literature, D-GPTScore is introduced as a human-aligned evaluation method that decomposes evaluation criteria into explicit aspects and scores each aspect with GPT-4o before aggregation [2509.03385]. In a separate, verification-oriented formulation that mirrors VERDI, “D-GPTScore” is presented as a single-call, post-hoc confidence estimator for structured LLM judges: it parses a verification trace into sub-checks, extracts structural signals, and combines them with Platt-scaled logistic regression to produce a calibrated confidence for a binary verdict [2605.11334]. Across both uses, the unifying principle is explicit decomposition of an otherwise monolithic judgment. ## 1. Definition and scope In the concept customization setting, D-GPTScore is formally motivated by the observation that existing automatic metrics for concept customization either focus on a single narrow aspect or provide only a coarse overall score, and as a result correlate poorly with human judgments. The method therefore makes the factors in evaluation explicit and re-aggregates them, using a Multimodal LLM as the judge [2509.03385]. The formalism defines text prompt elements $T=\{t_1,\dots,t_k\}$, reference images $I=\{i_1,\dots,i_l\}$, and a generative model $\theta$ producing a synthesized image $$ I_g=\theta(T,I). $$ Given a fixed set of $N$ evaluation aspects $A=\{a_1,\dots,a_N\}$, aspect-wise scores are obtained by querying a multimodal LLM $\phi$: $$ s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N. $$ The aspect scores are then aggregated into an overall score on a 1–10 scale: $$ s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N), $$ with the paper using simple averaging in practice: $$ \Phi(s_1\dots s_N)=\frac{1}{N}\sum_n s_n. $$ These definitions make decomposition the central architectural choice rather than a post-hoc analytical convenience [2509.03385]. In the verification-oriented formulation, the objective is different but structurally analogous: given a structured LLM-judge call of the form evidence plus generated claim leading to analysis trace and verdict, the goal is to compute a calibrated confidence score $p\in[0,1]$ for the binary verdict without extra LLM calls or token log-probabilities. Parallel to VERDI, this version of D-GPTScore decomposes the reasoning trace into claim extraction, per-claim verification, and aggregation, then derives structural signals from that single trace [2605.11334]. This suggests that D-GPTScore is best understood not as a single benchmark-specific metric, but as a general decomposition-and-aggregation pattern instantiated differently in multimodal evaluation and verification-based confidence estimation. ## 2. Aspect decomposition for concept customization For concept customization, D-GPTScore decomposes evaluation into 18 explicit aspects grouped into two categories: Concept Fidelity and Quality Assessment [2509.03385]. The aspect inventory is intended to cover factors that humans naturally consider during concept customization. ### Concept Fidelity (13 aspects) 1. Subject Type (object existence): Are the correct classes (man vs. woman, dog vs. cat) generated? 2. Quantity (object existence): Is the number of instances correct? 3. Camera Positioning (layout composition): Does the shot type (close-up, high angle) match the prompt? 4. Size/Scale (layout composition): Are relative and absolute sizes of subjects appropriate? 5. Color (object-level fidelity): Do colors match the reference images? 6. Subject Completeness (object-level fidelity): Are any body parts missing or spurious? 7. Proportions/Body Consistency (object-level fidelity): Are body proportions and limb poses natural? 8. Actions/Expressions (object-level fidelity): Are the described actions, poses, facial expressions correctly depicted? 9. Clothing/Attributes (object-level fidelity): Are outfits and accessories consistent with reference? 10. Facial Similarity/Features (object-level fidelity): Does the face preserve identity-defining features? 11. Surroundings (object-level fidelity): Does the environment match the prompt? 12. Human/Animal Interactions (multi-concept consistency): Are human-animal or person-person interactions natural? 13. Object Interactions (multi-concept consistency): Are interactions between objects (books, chairs) coherent? ### Quality Assessment (5 aspects) 14. Subject Deformation: Are subjects free of unnatural warping? 15. Surroundings Deformation: Are background elements free of artifacts like crooked lines? 16. Local Artifacts: Are there local glitches, noise, or unwanted patterns? 17. Detail/Sharpness: Are fine details (hands, faces) clearly rendered? 18. Style Consistency: Does the overall look match the requested style (e.g., realistic vs. anime)? Sections 3.4 and A.5 describe a bottom-up process in which an exhaustive list of possible human evaluation factors was pruned for redundancy and grouped into similar items, inspired by prior decomposed scoring work such as MetaMetrics and VideoScore [2509.03385]. An ablation reported in Table 9 shows that removing decomposition with Vanilla-GPT substantially drops correlation with human preference, which the paper presents as evidence that these aspects are necessary and sufficient [2509.03385]. A plausible implication is that the aspect inventory serves not only as a scoring interface for GPT-4o, but also as a theory of what constitutes human preference in concept customization under the benchmark’s task design. ## 3. Multimodal scoring workflow and aggregation The multimodal instantiation uses GPT-4o-2024-08-06 as $\phi$ [2509.03385]. Input construction is aspect-dependent: for each aspect, the prompt includes exactly the inputs required—generated image $I_g$ plus either $T$, $I$, or neither—so as not to overwhelm the LLM with irrelevant information [2509.03385]. The prompt templates are specialized by dependency type. If an aspect depends on $T$, as in Actions/Expressions, the model receives the prompt text and generated image together with an instruction of the form: > Task: … rate how well the generated image meets the following evaluation aspect… > Evaluation aspect: \ > Scoring example: \ > The text prompt: ‘…’ > \ > Score (1–5): If an aspect depends on reference images $I$, as in Color or Facial Similarity, the template omits $T$ and shows the reference images. For pure quality aspects such as Local Artifacts and Deformations, no $T$ or $I$ is provided; instead, the model is shown three crops—left, right, and full—to guard against hidden errors at image edges [2509.03385]. The output is an integer score from 1 to 5 for each aspect. The overall D-GPTScore is then formed by simple averaging of the 18 aspect-wise scores and mapped to a 1–10 scale through the aggregation function $\Phi$ as defined above [2509.03385]. An aggregation ablation compared linear regression with averaging and reported $r/\rho = 0.75 / 0.62$ for regression versus $0.78 / 0.69$ for averaging; the paper suggests regression may overfit with few models, while averaging works robustly out-of-the-box [2509.03385]. This design makes the scoring pipeline modular: aspect prompts, aspect-wise outputs, and final aggregation are cleanly separated. That separation is central to the method’s interpretability, because it permits aspect-level inspection in addition to a single scalar score. ## 4. Verification-style D-GPTScore and structural confidence signals In the verification-oriented formulation mirroring VERDI, D-GPTScore operates on a structured judge trace rather than an image-generation task [2605.11334]. The judge is prompted to perform three steps in a single forward pass: 1. ClaimExtraction: identify individual propositions $C=\{c_1,\dots,c_n\}$. 2. PerClaimAdjudication: for each $c_i$, compare against the evidence and label it, for example VERIFIED, FABRICATED, or NOT\_FOUND. 3. Aggregation: combine all per-claim labels into a final verdict $v\in\{\text{YES},\text{NO}\}$. The extraction pipeline is described procedurally: ```text function RunJudge(evidence, generated_claim): prompt ← build_decomposed_prompt(evidence, generated_claim) trace ← LLM(prompt) # single call claims ← parse_claims(trace) labels ← parse_labels(trace) # one label per claim verdict ← parse_verdict(trace) return (claims, labels, verdict, trace) ``` In practice, `parse_claims`, `parse_labels`, and `parse_verdict` are deterministic regex or NLI-based extractors that operate on the JSON-style trace [2605.11334]. Three structural signals are then defined. First, **Step-Verdict Alignment (SVA)** measures whether explicit local conclusions support the final verdict. With reasoning steps $s_1,\dots,s_n$, a subset $\mathcal S_{\mathrm{conc}}\subseteq\{1,\dots,n\}$ containing explicit local conclusions, and verdict $v$, the alignment function is $$ \mathrm{align}(s_i,v)= \begin{cases} 1 & \text{if step } s_i \text{ supports verdict } v,\\ 0 & \text{otherwise.} \end{cases} $$ Then $$ \mathrm{SVA} =\frac{\sum_{i\in\mathcal S_{\mathrm{conc}}}\mathrm{align}(s_i,v)} {|\mathcal S_{\mathrm{conc}}|}. $$ Second, **Claim-Level Margin (CLM)** measures how strongly claims lean toward one label. Encoding per-claim outcomes as $y_i\in\{0,1\}$, for example $y_i=1$ if $c_i$ is VERIFIED and $0$ otherwise, $$ \mathrm{CLM} =\frac{\max\bigl(\sum_{i=1}^m y_i,\; m-\sum_{i=1}^m y_i\bigr)}{m}. $$ Third, **Evidence Grounding Score (EGS)** measures how well quoted evidence spans in the trace match the source evidence. For quoted spans $e_1,\dots,e_k$ with lengths $\ell_j$ and indicator $\delta_j=1$ if grounded, else $0$, $$ \mathrm{EGS} =\frac{\sum_{j=1}^k \delta_j\,\ell_j}{\sum_{j=1}^k \ell_j}. $$ Hallucinated or ungrounded quotes drive EGS downward [2605.11334]. These features are combined through Platt-scaled logistic regression. With $x_1=\mathrm{SVA}$, $x_2=\mathrm{CLM}$, $x_3=\mathrm{EGS}$, and $\mathbf{x}=(x_1,x_2,x_3)$, $$ p(\mathbf x) =\frac{1}{1+\exp\bigl(-\bigl(a+b_1x_1+b_2x_2+b_3x_3\bigr)\bigr)}. $$ The scalar $a$ and weights $(b_1,b_2,b_3)$ are learned by minimizing the regularized negative log-likelihood on a small held-out calibration set, for example via 5-fold CV with $\ell_2$ penalty $\lambda=0.1$ [2605.11334]. The associated one-pass algorithm is: ```text function D_GPTScore(evidence, generated_claim): (claims, labels, verdict, trace) ← RunJudge(evidence, generated_claim) # 1. Compute SVA steps ← extract_conclusion_steps(trace) aligned ← count_alignments(steps, verdict) x1 ← aligned / |steps| # 2. Compute CLM y ← [encode(label) for label in labels] # 1=verified,0=not x2 ← max(sum(y), len(y)-sum(y)) / len(y) # 3. Compute EGS spans ← extract_quoted_spans(trace) weights ← [len(span) for span in spans] grounded ← [match(span, evidence) for span in spans] x3 ← dot(grounded, weights) / sum(weights) # 4. Standardize features, then z ← a + b1*x1 + b2*x2 + b3*x3 p ← 1/(1 + exp(-z)) return p end function ``` No extra LLM calls or log-probs are needed; everything is parsed from the single structured trace [2605.11334]. ## 5. Benchmarks and empirical performance For concept customization, D-GPTScore is evaluated on CC-AlignBench, a benchmark with 196 base prompts divided into three difficulty levels: Easy, Medium, and Hard [2509.03385]. Easy consists of 1 person, 13 single actions, and non-interactive settings. Medium consists of 2 persons performing the same 13 actions independently in non-interactive settings. Hard consists of 2 persons with 23 mutual interactions such as hugging and punching [2509.03385]. Each base prompt contains four elements—action, layout, expression, and surroundings—and produces five prompt variants: action-only, action+layout, action+expression, action+surroundings, and all four combined, giving a total of $196\times 5=980$ tasks [2509.03385]. Human preference annotation uses 12 expert annotators who scored 720 images, corresponding to 40 prompts × 3 difficulty levels × 6 generative models, on a 1–10 scale with reference to $T$ and $I$; scores were averaged across raters to form the ground-truth human preference per image [2509.03385]. Correlation with human scores is measured using Pearson’s $r$ and Spearman’s $\rho$. The reported overall results are: | Metric | Pearson’s $r$ | Spearman’s $\rho$ | |---|---:|---:| | ArcFace | 0.23 | 0.04 | | CLIP T2I | 0.29 | 0.42 | | CLIP T2T | 0.14 | 0.21 | | CLIP Aesthetic | 0.51 | 0.49 | | DINO | 0.10 | 0.04 | | D-GPTScore | 0.78 | 0.69 | Table 4 further reports benchmark scores on a 1–10 scale by difficulty level, predicted by D-GPTScore, showing the expected trend of performance dropping as tasks move from Easy to Hard [2509.03385]. Figure 3 provides aspect-wise outputs indicating model-specific strengths and weaknesses; for example, all models have low scores on complex body consistency, clothing attributes, and interaction aspects in the Hard setting, while OMG variants score higher on object interactions and local artifacts across difficulties [2509.03385]. In the verification-style formulation, the empirical results are framed in terms of confidence estimation AUROC and calibration [2605.11334]. On three public benchmarks—SummEval, FEVER, and SciFact—with GPT-4.1-mini, token log-prob baselines saturate at approximately AUROC $0.37$–$0.52$, trace-length yields $0.56$–$0.57$, and D-GPTScore, identified there as “VERDI LR,” yields $0.717$ on SummEval, $0.737$ on FEVER, and $0.910$ on SciFact [2605.11334]. On GPT-5.4-mini, where no log-probs are exposed, AUROC ranges from $0.662$ to $0.879$ across rubrics with Expected Calibration Error below $0.12$ [2605.11334]. On Qwen3.5-\{4B,9B,27B\}, where answer-token log-probs are anti-calibrated with AUROC $0.32$–$0.49$, D-GPTScore achieves $0.56$–$0.70$, corresponding to a $+0.08$ to $+0.31$ improvement [2605.11334]. Taken together, these results show strong empirical performance in two distinct tasks: ranking generated images by human preference and estimating whether an LLM judge’s own verdict should be trusted. ## 6. Interpretation, ablations, and limitations The concept customization paper includes several ablations clarifying which components of D-GPTScore matter most [2509.03385]. Removing decomposition and using Vanilla-GPT yields overall $r/\rho = 0.67 / 0.58$ instead of $0.78 / 0.69$. Replacing full GPT-4o with GPT-4o mini reduces overall correlation to $0.68 / 0.59$. Replacing averaging with linear regression yields $0.75 / 0.62$ instead of $0.78 / 0.69$ [2509.03385]. These results support the paper’s claim that explicit aspect decomposition and strong multimodal judging are important contributors to human alignment. The verification-style formulation identifies distinct failure modes [2605.11334]. Genuine reasoning errors in which all sub-checks align internally, so that $\mathrm{SVA}\approx 1.0$, but the model misinterprets evidence, can evade detection. Traces with minimal structure, such as style or bias checks, yield weaker signals, and the summary states that surface-feature-only variants may be needed [2605.11334]. A common misconception would be to treat decomposition as merely adding more prompts or more model calls. In the verification-oriented version, decomposition is explicitly single-call and post-hoc, extracting confidence from a structured trace already produced by the judge [2605.11334]. In the concept customization version, decomposition does involve aspect-wise assessments, but the central claim is not efficiency; it is better alignment with human preference by forcing explicit consideration of fine-grained factors [2509.03385]. Another misconception would be to assume that D-GPTScore has a unique fixed mathematical form across applications. The data instead indicate two distinct instantiations: one based on aspect-wise multimodal scoring and averaging, and another based on structural trace features and Platt-scaled logistic regression [2509.03385][2605.11334]. This suggests that “D-GPTScore” functions as a family resemblance term for decomposition-based judge scoring rather than a single canonical equation. ## 7. Extensions and research significance The concept customization work positions D-GPTScore as a human-aligned metric for single- and multi-concept tasks and states that it establishes a new standard for evaluating concept customization while highlighting key challenges for future research [2509.03385]. The benchmark and associated materials are publicly released, and the stated future directions include extending the dataset beyond humans to animals and rigid objects, handling models that consume additional optional inputs such as pose or sketch by expanding the aspect set and input handling for $\phi$, and exploring learnable aggregation such as regression or light neural nets as more generative models emerge [2509.03385]. The verification-oriented formulation outlines extensions beyond verification: decomposing pairwise preference judgments into sub-checks on each alternative; verifying topical consistency or factuality of generated segments for open-ended generation; and adding sub-checks such as rhetorical coherence, sentiment matching, or external-knowledge retrieval checks, each yielding new structural signals for the logistic model [2605.11334]. It also reports that a 33M-parameter NLI model provides a scalable alternative to regex extraction, and that cross-model transfer reaches AUROC $0.66$–$0.69$ [2605.11334]. These extensions indicate a broader methodological significance. D-GPTScore exemplifies a design pattern in which evaluation quality is improved by forcing a judge to expose intermediate structure and then scoring that structure rather than relying solely on a single undifferentiated judgment. In concept customization, the structure is semantic and perceptual aspect decomposition; in verification-based confidence estimation, it is decomposition into claim extraction, per-claim adjudication, and verdict aggregation. Across both cases, the central methodological claim is that decomposition yields signals that are more informative than coarse end-to-end scores or token-probability heuristics alone [2509.03385][2605.11334]. Source: https://www.emergentmind.com/topics/decomposed-gpt-score-d-gptscore