Decomposed GPT Score (D-GPTScore)

• D-GPTScore is a decomposition-based scoring framework that splits LLM judgments into fine-grained sub-assessments and re-aggregates them to better align with human preferences.
• It uses GPT-4o to evaluate explicit aspects such as concept fidelity and quality in multimodal concept customization, achieving high correlation with human annotations.
• In verification settings, D-GPTScore leverages a single-call structured trace with claim extraction and logistic regression to deliver calibrated confidence scores for binary verdicts.

Decomposed GPT Score (D-GPTScore) denotes a decomposition-based scoring paradigm in which a LLM or multimodal LLM evaluates an output by splitting the judgment into finer-grained sub-assessments and then aggregating them into a final score. In the concept customization literature, D-GPTScore is introduced as a human-aligned evaluation method that decomposes evaluation criteria into explicit aspects and scores each aspect with GPT-4o before aggregation (Ishikawa et al., 3 Sep 2025). In a separate, verification-oriented formulation that mirrors VERDI, “D-GPTScore” is presented as a single-call, post-hoc confidence estimator for structured LLM judges: it parses a verification trace into sub-checks, extracts structural signals, and combines them with Platt-scaled logistic regression to produce a calibrated confidence for a binary verdict (Qi et al., 11 May 2026). Across both uses, the unifying principle is explicit decomposition of an otherwise monolithic judgment.

1. Definition and scope

In the concept customization setting, D-GPTScore is formally motivated by the observation that existing automatic metrics for concept customization either focus on a single narrow aspect or provide only a coarse overall score, and as a result correlate poorly with human judgments. The method therefore makes the factors in evaluation explicit and re-aggregates them, using a Multimodal LLM as the judge (Ishikawa et al., 3 Sep 2025).

The formalism defines text prompt elements $T=\{t_1,\dots,t_k\}$, reference images $I=\{i_1,\dots,i_l\}$, and a generative model $\theta$ producing a synthesized image

$I_g=\theta(T,I).$

Given a fixed set of $N$ evaluation aspects $A=\{a_1,\dots,a_N\}$, aspect-wise scores are obtained by querying a multimodal LLM $\phi$:

$s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$

The aspect scores are then aggregated into an overall score on a 1–10 scale:

$s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N),$

with the paper using simple averaging in practice:

$\Phi(s_1\dots s_N)=\frac{1}{N}\sum_n s_n.$

These definitions make decomposition the central architectural choice rather than a post-hoc analytical convenience (Ishikawa et al., 3 Sep 2025).

In the verification-oriented formulation, the objective is different but structurally analogous: given a structured LLM-judge call of the form evidence plus generated claim leading to analysis trace and verdict, the goal is to compute a calibrated confidence score $I=\{i_1,\dots,i_l\}$0 for the binary verdict without extra LLM calls or token log-probabilities. Parallel to VERDI, this version of D-GPTScore decomposes the reasoning trace into claim extraction, per-claim verification, and aggregation, then derives structural signals from that single trace (Qi et al., 11 May 2026).

This suggests that D-GPTScore is best understood not as a single benchmark-specific metric, but as a general decomposition-and-aggregation pattern instantiated differently in multimodal evaluation and verification-based confidence estimation.

2. Aspect decomposition for concept customization

For concept customization, D-GPTScore decomposes evaluation into 18 explicit aspects grouped into two categories: Concept Fidelity and Quality Assessment (Ishikawa et al., 3 Sep 2025). The aspect inventory is intended to cover factors that humans naturally consider during concept customization.

Concept Fidelity (13 aspects)

1. Subject Type (object existence): Are the correct classes (man vs. woman, dog vs. cat) generated?
2. Quantity (object existence): Is the number of instances correct?
3. Camera Positioning (layout composition): Does the shot type (close-up, high angle) match the prompt?
4. Size/Scale (layout composition): Are relative and absolute sizes of subjects appropriate?
5. Color (object-level fidelity): Do colors match the reference images?
6. Subject Completeness (object-level fidelity): Are any body parts missing or spurious?
7. Proportions/Body Consistency (object-level fidelity): Are body proportions and limb poses natural?
8. Actions/Expressions (object-level fidelity): Are the described actions, poses, facial expressions correctly depicted?
9. Clothing/Attributes (object-level fidelity): Are outfits and accessories consistent with reference?
10. Facial Similarity/Features (object-level fidelity): Does the face preserve identity-defining features?
11. Surroundings (object-level fidelity): Does the environment match the prompt?
12. Human/Animal Interactions (multi-concept consistency): Are human-animal or person-person interactions natural?
13. Object Interactions (multi-concept consistency): Are interactions between objects (books, chairs) coherent?

Quality Assessment (5 aspects)

1. Subject Deformation: Are subjects free of unnatural warping?
2. Surroundings Deformation: Are background elements free of artifacts like crooked lines?
3. Local Artifacts: Are there local glitches, noise, or unwanted patterns?
4. Detail/Sharpness: Are fine details (hands, faces) clearly rendered?
5. Style Consistency: Does the overall look match the requested style (e.g., realistic vs. anime)?

Sections 3.4 and A.5 describe a bottom-up process in which an exhaustive list of possible human evaluation factors was pruned for redundancy and grouped into similar items, inspired by prior decomposed scoring work such as MetaMetrics and VideoScore (Ishikawa et al., 3 Sep 2025). An ablation reported in Table 9 shows that removing decomposition with Vanilla-GPT substantially drops correlation with human preference, which the paper presents as evidence that these aspects are necessary and sufficient (Ishikawa et al., 3 Sep 2025).

A plausible implication is that the aspect inventory serves not only as a scoring interface for GPT-4o, but also as a theory of what constitutes human preference in concept customization under the benchmark’s task design.

3. Multimodal scoring workflow and aggregation

The multimodal instantiation uses GPT-4o-2024-08-06 as $I=\{i_1,\dots,i_l\}$1 (Ishikawa et al., 3 Sep 2025). Input construction is aspect-dependent: for each aspect, the prompt includes exactly the inputs required—generated image $I=\{i_1,\dots,i_l\}$2 plus either $I=\{i_1,\dots,i_l\}$3, $I=\{i_1,\dots,i_l\}$4, or neither—so as not to overwhelm the LLM with irrelevant information (Ishikawa et al., 3 Sep 2025).

The prompt templates are specialized by dependency type. If an aspect depends on $I=\{i_1,\dots,i_l\}$5, as in Actions/Expressions, the model receives the prompt text and generated image together with an instruction of the form:

Task: … rate how well the generated image meets the following evaluation aspect… Evaluation aspect: <aspect description> Scoring example: <example> The text prompt: ‘…’ <Generated image> Score (1–5):

If an aspect depends on reference images $I=\{i_1,\dots,i_l\}$6, as in Color or Facial Similarity, the template omits $I=\{i_1,\dots,i_l\}$7 and shows the reference images. For pure quality aspects such as Local Artifacts and Deformations, no $I=\{i_1,\dots,i_l\}$8 or $I=\{i_1,\dots,i_l\}$9 is provided; instead, the model is shown three crops—left, right, and full—to guard against hidden errors at image edges (Ishikawa et al., 3 Sep 2025). The output is an integer score from 1 to 5 for each aspect.

The overall D-GPTScore is then formed by simple averaging of the 18 aspect-wise scores and mapped to a 1–10 scale through the aggregation function $\theta$0 as defined above (Ishikawa et al., 3 Sep 2025). An aggregation ablation compared linear regression with averaging and reported $\theta$1 for regression versus $\theta$2 for averaging; the paper suggests regression may overfit with few models, while averaging works robustly out-of-the-box (Ishikawa et al., 3 Sep 2025).

This design makes the scoring pipeline modular: aspect prompts, aspect-wise outputs, and final aggregation are cleanly separated. That separation is central to the method’s interpretability, because it permits aspect-level inspection in addition to a single scalar score.

4. Verification-style D-GPTScore and structural confidence signals

In the verification-oriented formulation mirroring VERDI, D-GPTScore operates on a structured judge trace rather than an image-generation task (Qi et al., 11 May 2026). The judge is prompted to perform three steps in a single forward pass:

1. ClaimExtraction: identify individual propositions $\theta$3.
2. PerClaimAdjudication: for each $\theta$4, compare against the evidence and label it, for example VERIFIED, FABRICATED, or NOT_FOUND.
3. Aggregation: combine all per-claim labels into a final verdict $\theta$5.

The extraction pipeline is described procedurally:

$s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N),$2

In practice, parse_claims, parse_labels, and parse_verdict are deterministic regex or NLI-based extractors that operate on the JSON-style trace (Qi et al., 11 May 2026).

Three structural signals are then defined.

First, Step-Verdict Alignment (SVA) measures whether explicit local conclusions support the final verdict. With reasoning steps $\theta$6, a subset $\theta$7 containing explicit local conclusions, and verdict $\theta$8, the alignment function is

$\theta$9

Then

$I_g=\theta(T,I).$0

Second, Claim-Level Margin (CLM) measures how strongly claims lean toward one label. Encoding per-claim outcomes as $I_g=\theta(T,I).$1, for example $I_g=\theta(T,I).$2 if $I_g=\theta(T,I).$3 is VERIFIED and $I_g=\theta(T,I).$4 otherwise,

$I_g=\theta(T,I).$5

Third, Evidence Grounding Score (EGS) measures how well quoted evidence spans in the trace match the source evidence. For quoted spans $I_g=\theta(T,I).$6 with lengths $I_g=\theta(T,I).$7 and indicator $I_g=\theta(T,I).$8 if grounded, else $I_g=\theta(T,I).$9,

$N$0

Hallucinated or ungrounded quotes drive EGS downward (Qi et al., 11 May 2026).

These features are combined through Platt-scaled logistic regression. With $N$1, $N$2, $N$3, and $N$4,

$N$5

The scalar $N$6 and weights $N$7 are learned by minimizing the regularized negative log-likelihood on a small held-out calibration set, for example via 5-fold CV with $N$8 penalty $N$9 (Qi et al., 11 May 2026).

The associated one-pass algorithm is:

$s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N),$3

No extra LLM calls or log-probs are needed; everything is parsed from the single structured trace (Qi et al., 11 May 2026).

5. Benchmarks and empirical performance

For concept customization, D-GPTScore is evaluated on CC-AlignBench, a benchmark with 196 base prompts divided into three difficulty levels: Easy, Medium, and Hard (Ishikawa et al., 3 Sep 2025). Easy consists of 1 person, 13 single actions, and non-interactive settings. Medium consists of 2 persons performing the same 13 actions independently in non-interactive settings. Hard consists of 2 persons with 23 mutual interactions such as hugging and punching (Ishikawa et al., 3 Sep 2025). Each base prompt contains four elements—action, layout, expression, and surroundings—and produces five prompt variants: action-only, action+layout, action+expression, action+surroundings, and all four combined, giving a total of $A=\{a_1,\dots,a_N\}$0 tasks (Ishikawa et al., 3 Sep 2025).

Human preference annotation uses 12 expert annotators who scored 720 images, corresponding to 40 prompts × 3 difficulty levels × 6 generative models, on a 1–10 scale with reference to $A=\{a_1,\dots,a_N\}$1 and $A=\{a_1,\dots,a_N\}$2; scores were averaged across raters to form the ground-truth human preference per image (Ishikawa et al., 3 Sep 2025). Correlation with human scores is measured using Pearson’s $A=\{a_1,\dots,a_N\}$3 and Spearman’s $A=\{a_1,\dots,a_N\}$4.

The reported overall results are:

Metric	Pearson’s $A=\{a_1,\dots,a_N\}$5	Spearman’s $A=\{a_1,\dots,a_N\}$6
ArcFace	0.23	0.04
CLIP T2I	0.29	0.42
CLIP T2T	0.14	0.21
CLIP Aesthetic	0.51	0.49
DINO	0.10	0.04
D-GPTScore	0.78	0.69

Table 4 further reports benchmark scores on a 1–10 scale by difficulty level, predicted by D-GPTScore, showing the expected trend of performance dropping as tasks move from Easy to Hard (Ishikawa et al., 3 Sep 2025). Figure 1 provides aspect-wise outputs indicating model-specific strengths and weaknesses; for example, all models have low scores on complex body consistency, clothing attributes, and interaction aspects in the Hard setting, while OMG variants score higher on object interactions and local artifacts across difficulties (Ishikawa et al., 3 Sep 2025).

In the verification-style formulation, the empirical results are framed in terms of confidence estimation AUROC and calibration (Qi et al., 11 May 2026). On three public benchmarks—SummEval, FEVER, and SciFact—with GPT-4.1-mini, token log-prob baselines saturate at approximately AUROC $A=\{a_1,\dots,a_N\}$7–$A=\{a_1,\dots,a_N\}$8, trace-length yields $A=\{a_1,\dots,a_N\}$9–$\phi$0, and D-GPTScore, identified there as “VERDI LR,” yields $\phi$1 on SummEval, $\phi$2 on FEVER, and $\phi$3 on SciFact (Qi et al., 11 May 2026). On GPT-5.4-mini, where no log-probs are exposed, AUROC ranges from $\phi$4 to $\phi$5 across rubrics with Expected Calibration Error below $\phi$6 (Qi et al., 11 May 2026). On Qwen3.5-{4B,9B,27B}, where answer-token log-probs are anti-calibrated with AUROC $\phi$7–$\phi$8, D-GPTScore achieves $\phi$9–$s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$0, corresponding to a $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$1 to $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$2 improvement (Qi et al., 11 May 2026).

Taken together, these results show strong empirical performance in two distinct tasks: ranking generated images by human preference and estimating whether an LLM judge’s own verdict should be trusted.

6. Interpretation, ablations, and limitations

The concept customization paper includes several ablations clarifying which components of D-GPTScore matter most (Ishikawa et al., 3 Sep 2025). Removing decomposition and using Vanilla-GPT yields overall $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$3 instead of $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$4. Replacing full GPT-4o with GPT-4o mini reduces overall correlation to $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$5. Replacing averaging with linear regression yields $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$6 instead of $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$7 (Ishikawa et al., 3 Sep 2025). These results support the paper’s claim that explicit aspect decomposition and strong multimodal judging are important contributors to human alignment.

The verification-style formulation identifies distinct failure modes (Qi et al., 11 May 2026). Genuine reasoning errors in which all sub-checks align internally, so that $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$8, but the model misinterprets evidence, can evade detection. Traces with minimal structure, such as style or bias checks, yield weaker signals, and the summary states that surface-feature-only variants may be needed (Qi et al., 11 May 2026).

A common misconception would be to treat decomposition as merely adding more prompts or more model calls. In the verification-oriented version, decomposition is explicitly single-call and post-hoc, extracting confidence from a structured trace already produced by the judge (Qi et al., 11 May 2026). In the concept customization version, decomposition does involve aspect-wise assessments, but the central claim is not efficiency; it is better alignment with human preference by forcing explicit consideration of fine-grained factors (Ishikawa et al., 3 Sep 2025).

Another misconception would be to assume that D-GPTScore has a unique fixed mathematical form across applications. The data instead indicate two distinct instantiations: one based on aspect-wise multimodal scoring and averaging, and another based on structural trace features and Platt-scaled logistic regression (Ishikawa et al., 3 Sep 2025, Qi et al., 11 May 2026). This suggests that “D-GPTScore” functions as a family resemblance term for decomposition-based judge scoring rather than a single canonical equation.

7. Extensions and research significance

The concept customization work positions D-GPTScore as a human-aligned metric for single- and multi-concept tasks and states that it establishes a new standard for evaluating concept customization while highlighting key challenges for future research (Ishikawa et al., 3 Sep 2025). The benchmark and associated materials are publicly released, and the stated future directions include extending the dataset beyond humans to animals and rigid objects, handling models that consume additional optional inputs such as pose or sketch by expanding the aspect set and input handling for $s_n=\phi(a_n,I_g,T,I)\quad \text{for } n=1\dots N.$9, and exploring learnable aggregation such as regression or light neural nets as more generative models emerge (Ishikawa et al., 3 Sep 2025).

The verification-oriented formulation outlines extensions beyond verification: decomposing pairwise preference judgments into sub-checks on each alternative; verifying topical consistency or factuality of generated segments for open-ended generation; and adding sub-checks such as rhetorical coherence, sentiment matching, or external-knowledge retrieval checks, each yielding new structural signals for the logistic model (Qi et al., 11 May 2026). It also reports that a 33M-parameter NLI model provides a scalable alternative to regex extraction, and that cross-model transfer reaches AUROC $s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N),$0–$s_{\mathrm{overall}}=\Phi(s_1,s_2,\dots,s_N),$1 (Qi et al., 11 May 2026).

These extensions indicate a broader methodological significance. D-GPTScore exemplifies a design pattern in which evaluation quality is improved by forcing a judge to expose intermediate structure and then scoring that structure rather than relying solely on a single undifferentiated judgment. In concept customization, the structure is semantic and perceptual aspect decomposition; in verification-based confidence estimation, it is decomposition into claim extraction, per-claim adjudication, and verdict aggregation. Across both cases, the central methodological claim is that decomposition yields signals that are more informative than coarse end-to-end scores or token-probability heuristics alone (Ishikawa et al., 3 Sep 2025, Qi et al., 11 May 2026).