CC-AlignBench: Preference-Based Concept Customization

• The paper introduces CC-AlignBench to evaluate generative models on preserving concept identity and adhering to prompt constraints using a decomposed evaluation approach.
• It assesses both single-concept and multi-concept image generation through staged difficulties that test action execution, layout, and interaction fidelity.
• It employs the 18-aspect D-GPTScore with an MLLM to achieve closer alignment with human preferences than traditional single-score metrics.

Human Preference-Aligned Concept Customization Benchmark (CC-AlignBench) is a benchmark for evaluating concept customization under human preference, especially in settings where a generative model must preserve concept identity or style from reference images while following prompt constraints, including actions, layout, expressions, surroundings, and, in multi-concept cases, inter-concept interactions. In the benchmark formulation, a generated image $I_g$ is produced from a text prompt $T$ and a set of concept reference images $I$ by a model $\theta$, written as $I_g = \theta(T, I)$. CC-AlignBench is paired with Decomposed GPT Score (D-GPTScore), an evaluation method that decomposes assessment into fine-grained aspects and scores them with a Multimodal LLM (MLLM), with the stated goal of achieving closer alignment with human preferences than single-score or narrowly scoped metrics (Ishikawa et al., 3 Sep 2025).

1. Task definition and motivation

Concept customization is defined as a setting in which a model receives a small set of reference images for each concept together with a user text prompt, and must generate an image that both preserves concept identity or style and follows prompt conditions. In CC-AlignBench, this task is centered on humans and is designed to test both single-concept and multi-concept generation. The benchmark emphasizes that single-concept evaluation already requires judging identity preservation, attribute matching, pose or action fidelity, layout, and overall image quality, while multi-concept evaluation is considerably harder because it also requires assessment of interactions between concepts, such as physical contact, relative layout, joint actions, occlusions, size relations, camera vantage, and prompt-specific conditions like expressions, surroundings, and style (Ishikawa et al., 3 Sep 2025).

The benchmark is motivated by a claimed mismatch between human judgments and existing automated metrics. The benchmark description groups prior metrics into several insufficient classes. Object and layout metrics such as Counting, SOA-I, Compositions, and YOLO Score overlook verb or action fidelity. CLIP-based and DINO features provide coarse similarity and may miss subtle identity or action correctness. Image quality and aesthetic metrics such as IS, GIQA/QS, and CLIP Aesthetic correlate with perceived quality but not with prompt or concept fidelity. Preference or VQA-based scores such as ImageReward and VQA Score focus on text-to-image rather than customization and lack explicit multi-concept checks. GPT-4V-based single-score evaluators are described as missing multi-concept interactions when not decomposed. The benchmark therefore frames decomposed evaluation as necessary for human-aligned scoring in concept customization (Ishikawa et al., 3 Sep 2025).

A common misconception is that concept customization can be adequately assessed by a single global similarity score. The benchmark explicitly rejects that premise. Its central claim is that action fidelity, identity preservation, and interaction correctness are separable dimensions whose omission leads to low correlation with human preferences, and that a decomposed evaluation protocol addresses this failure mode (Ishikawa et al., 3 Sep 2025).

2. Benchmark composition and staged difficulty

CC-AlignBench focuses on humans, specifically to stress-test action diversity and multi-person interactions. The benchmark contains 196 base prompts divided into three difficulty levels: 52 Easy, 52 Medium, and 92 Hard. Each base prompt has five prompt variants, yielding $196 \times 5 = 980$ prompt variants in total. The five prompt types are action-only $\{t_{\text{act}}\}$, action+layout $\{t_{\text{act}}, t_{\text{lt}}\}$, action+expression $\{t_{\text{act}}, t_{\text{exp}}\}$, action+surroundings $\{t_{\text{act}}, t_{\text{surr}}\}$, and all $T$0 (Ishikawa et al., 3 Sep 2025).

The benchmark stages difficulty by increasing the number of entities and the complexity of the required relations.

Stage	Scenario	Prompt variants
Easy	Single person acting	260
Medium	Two persons with independent non-interactive actions	260
Hard	Two persons with mutual interaction actions	460

Easy consists of single-person prompts with 13 individual actions and one person per prompt. Medium consists of two-person scenes in which each person performs an independent non-interactive action; it also uses 13 actions. Hard consists of two-person mutual interaction actions and uses 23 interaction types. Examples of individual actions include standing, walking, running, waving, clapping, hands in pockets, jumping, checking wristwatch, crossing arms, kneeling, squatting, punching, and shrugging. Examples of mutual actions include hugging, shaking hands, giving or exchanging a book, whispering, taking a picture of the other, clinking glasses, carrying a box together, high-five, following down a street, supporting as they walk, grabbing a book, stepping on foot, patting on back, pointing, punching, kicking, pushing, and knocking into (Ishikawa et al., 3 Sep 2025).

Prompt construction is explicitly factorized into four elements: action, layout, expression, and surroundings. The benchmark provides concrete prompt realizations at each conditioning level. For the mutual action “putting arm around shoulder,” the action-only prompt is “A photo of a woman putting her arm around a man's shoulder, Ultra HD quality.” The fully conditioned prompt is “A high angle shot of a woman putting her arm around a man's shoulder, standing close, both looking amused, in an open green park, Ultra HD quality.” This staged prompting makes it possible to vary conditioning complexity without changing the core action (Ishikawa et al., 3 Sep 2025).

The concept references are limited to two human concepts, one woman and one man. There are 20 generated reference images per concept, with varied views including close-up or long shot and frontal or profile, for a total of 40 reference images. For methods requiring a single reference image, the benchmark convention is to use the same full-body image consistently (Ishikawa et al., 3 Sep 2025).

The scope of each stage is defined operationally. Easy evaluates single-concept identity fidelity and action execution in straightforward scenes. Medium evaluates two-person scenes with independent actions, focusing on spatial relations, relative sizing, camera or layout fidelity, and per-concept identity consistency without explicit interactions. Hard evaluates mutual interactions and emphasizes interaction correctness, contact plausibility, occlusions, spatial reasoning, and simultaneous identity preservation (Ishikawa et al., 3 Sep 2025).

3. Human preference annotation protocol

The benchmark includes human preference annotations from 12 expert annotators, of whom 5 were female and 7 male. Annotators scored generated images on a 1–10 scale using both the text prompt and the reference images as context. The instructions were deliberately minimal so as to reflect natural human judgment. Scores were averaged across annotators to obtain a single preference score per image (Ishikawa et al., 3 Sep 2025).

The annotation workload covered 40 prompts per difficulty level across 6 models, producing 720 generated images evaluated overall. Approximate annotation time was 2 hours per annotator, and compensation was $48 per annotator. Inter-rater reliability metrics, such as ICC, were not reported. This omission is important because it limits direct assessment of annotator consistency, even though the benchmark uses averaged scores as its reference target (Ishikawa et al., 3 Sep 2025).

CC-AlignBench is presented as an evaluation benchmark rather than a benchmark with a mandated train/test partition. No training/test split is mandated. The dataset license is CC BY-NC 4.0. Benchmark materials, evaluation code, and related resources are hosted at https://github.com/ReinaIshikawa/D-GPTScore (Ishikawa et al., 3 Sep 2025).

A further point of interpretation concerns what the human score represents. Because annotators are given both prompt and references and only minimal instructions, the benchmark’s target is not a narrowly operationalized submetric such as face similarity or text-image matching. Instead, the averaged score functions as a composite preference judgment over prompt fidelity, concept fidelity, interaction plausibility, and image quality. This interpretation is consistent with the benchmark’s emphasis on human-aligned evaluation rather than a single formal criterion (Ishikawa et al., 3 Sep 2025).

4. Decomposed GPT Score (D-GPTScore)

D-GPTScore is the benchmark’s principal automated evaluator. Its core idea is to use an MLLM to score fine-grained aspects of concept customization independently and then aggregate those aspect-wise scores into an overall score aligned with human preferences. Let the aspect set be $T$1 with $T$2. For aspect $T$3, the score is obtained as

$T$4

where $T$5 is the MLLM. The overall score is then

$T$6

with the default aggregation

$T$7

The text states that $T$8 lies on $T$9, but the implementation averages aspect scores that are in $I$0; the reported results are given on this implemented basis (Ishikawa et al., 3 Sep 2025).

The 18 aspects are divided into 13 Concept Fidelity aspects and 5 Quality Assessment aspects. The Concept Fidelity aspects are Subject Type; Quantity; Subject & Camera Positioning; Size & Scale; Color; Subject Completeness; Proportions & Body Consistency; Actions & Expressions; Clothing & Attributes; Facial Similarity & Features; Surroundings; Human & Animal Interactions; and Object Interactions. The Quality Assessment aspects are Subject Deformation; Surroundings Deformation; Local Artifacts; Detail & Sharpness; and Style Consistency (Ishikawa et al., 3 Sep 2025).

This decomposition has a specific methodological role. Human & Animal Interactions and Object Interactions explicitly address multi-entity relations such as hugging versus handshaking. No extra pairwise penalty terms or cross-aspect formulas are introduced beyond aspect-wise scoring; interaction handling is achieved by focused aspect prompts rather than a separate relational loss or interaction-specific aggregation rule (Ishikawa et al., 3 Sep 2025).

The main evaluator used for results is GPT-4o (gpt-4o-2024-08-06). GPT-4o mini is also evaluated as a cost-sensitive alternative. The MLLM receives the generated image and selectively the text prompt and/or reference image set depending on the aspect. Prompt-fidelity aspects such as Actions & Expressions and Surroundings use $I$1 together with $I$2. Identity or attribute fidelity aspects such as Color, Facial Similarity & Features, and Clothing & Attributes use $I$3 together with $I$4. Artifact and deformation assessments use $I$5 only, plus crops. All images are resized to $I$6 before submission, and for deformation or artifact aspects the evaluator also receives two square crops at 50% of original size, taken from the left and right halves and vertically centered, to reveal local issues (Ishikawa et al., 3 Sep 2025).

Aspect prompts request integer or real-valued scores in $I$7 and include a task instruction, relevant inputs, and a scoring example. Temperature and sampling settings are not specified. Average token usage per generated image is reported as 12,091 input tokens and 96 output tokens for GPT-4o, and 299,672 input tokens and 104 output tokens for GPT-4o mini (Ishikawa et al., 3 Sep 2025).

The benchmark also evaluates alternative aggregation schemes. Linear regression over aspect scores is tested with a leave-one-out scheme to avoid leakage when benchmarking models, but it slightly underperforms simple averaging. No normalization is applied in the base D-GPTScore pipeline. In an extension termed Ours++, the 18 aspect scores are combined with six external metrics—ArcFace, CLIP T2T, CLIP T2I, CLIP Aesthetic, DINO, and Vanilla GPT—after min-max normalization to $I$8 (Ishikawa et al., 3 Sep 2025).

5. Correlation with human preferences and benchmark findings

The benchmark evaluates automated metrics against mean human ratings using Pearson’s correlation coefficient $I$9 and Spearman’s rank correlation $\theta$0, with

$\theta$1

Kendall’s $\theta$2 and p-values are not reported (Ishikawa et al., 3 Sep 2025).

The main overall result is that D-GPTScore achieves substantially higher correlation with human preferences than the reported baselines.

Metric	Pearson $\theta$3	Spearman $\theta$4
D-GPTScore (Ours)	0.78	0.69
ArcFace	0.23	0.04
CLIP T2I	0.29	0.42
CLIP T2T	0.14	0.21
CLIP Aesthetic	0.51	0.49
DINO	0.10	0.04

The benchmark authors describe $\theta$5 as strong, and D-GPTScore exceeds that threshold while all other listed metrics remain lower overall. This is used to support the claim that decomposing evaluation into explicit aspects improves human alignment in concept customization (Ishikawa et al., 3 Sep 2025).

Per-model correlations also favor D-GPTScore across all six benchmarked generators. For CustomDiffusion, D-GPTScore reaches 0.80 Pearson and 0.54 Spearman. For OMG+LoRA it reaches 0.66 and 0.34; for OMG+InstantID, 0.70 and 0.47; for FastComposer, 0.64 and 0.46; for Mix-of-Show, 0.65 and 0.44; and for DreamBooth, 0.64 and 0.38. These values remain consistently above the reported baselines for each model family (Ishikawa et al., 3 Sep 2025).

Ablations reinforce the role of decomposition. Direct single-score prompting, called Vanilla-GPT, underperforms decomposed scoring. GPT-4o mini reduces overall correlation by 0.10 in both Pearson and Spearman relative to GPT-4o, yielding $\theta$6 and $\theta$7, though it still exceeds traditional metrics. Linear regression aggregation yields $\theta$8 and $\theta$9, slightly below simple averaging. Ours++ modestly improves alignment to $I_g = \theta(T, I)$0 and $I_g = \theta(T, I)$1 under average aggregation, while Ours++ with linear regression gives $I_g = \theta(T, I)$2 and $I_g = \theta(T, I)$3 (Ishikawa et al., 3 Sep 2025).

The benchmark also exhibits intended stage-wise difficulty scaling. D-GPTScore drops from Easy to Hard for all reported models, indicating that interaction complexity systematically degrades performance.

Model	Overall	Easy / Medium / Hard
CustomDiffusion	4.74	5.64 / 4.62 / 4.30
OMG+LoRA	6.81	7.28 / 7.05 / 6.41
OMG+InstantID	6.52	7.13 / 6.64 / 6.10
FastComposer	4.72	5.21 / 4.72 / 4.45
Mix-of-Show	5.19	5.75 / 5.27 / 4.82
DreamBooth	5.09	5.53 / 5.01 / 4.87

Aspect-wise analysis shows that all models struggle with complex actions and interactions. The object-fidelity-related aspects that degrade most are Color, Facial Similarity & Features, Clothing & Attributes, Actions & Expressions, and Human & Animal Interactions. OMG-based methods are reported to score consistently higher on Proportions & Body Consistency and Object Interactions, and to suffer fewer Local Artifacts (Ishikawa et al., 3 Sep 2025).

6. Limitations, biases, and relation to broader human-preference modeling

CC-AlignBench has several explicit limitations. Its concepts are limited to two human identities; stationary objects and animals are not yet included, although the metric is described as applicable to them. The benchmark standardizes inputs to text and reference images and excludes optional controls used by some generation methods, such as pose or sketches, in order to preserve comparability and manage MLLM costs. D-GPTScore depends on an MLLM, and GPT-4o provides the strongest alignment at the expense of higher API usage. GPT-4o mini is identified as a viable but weaker alternative. Regression-based aggregators are described as more vulnerable to distribution shifts across models, whereas uniform averaging is treated as more robust in the present setting (Ishikawa et al., 3 Sep 2025).

The benchmark also discusses ethical considerations. Reference images are AI-generated to avoid portrait-rights issues. Offensive content is removed, although some action terms such as “hit” and “kick” are retained to diversify mutual interactions. The benchmark materials are intended for research on alignment and controllability under CC BY-NC 4.0, and the benchmark advises against generating harmful content, especially involving real individuals (Ishikawa et al., 3 Sep 2025).

A separate line of work on human preference decomposition provides a broader interpretive context for CC-AlignBench. “Learning a Canonical Basis of Human Preferences from Binary Ratings” identifies 21 preference categories—Clarity, Thoroughness, Accuracy, Concise, Relevance, Engagement, Innovation, Practicality, Informative, Diversity, Comprehension, Organization, Follows Instructions, Customization, Concentration, Helpfulness, Humor, Context, Environment, Direction, and Efficiency—and frames them as a low-rank, interpretable canonical basis that covers more than 89% of the 4,469 unique extracted preferences, with human MMC validation yielding $I_g = \theta(T, I)$4 and thus $I_g = \theta(T, I)$5 (Vodrahalli et al., 31 Mar 2025).

That framework is not the same as the 18-aspect D-GPTScore used in CC-AlignBench, but it is explicitly presented as adaptable to designing and evaluating a Human Preference-Aligned Concept Customization Benchmark. This suggests a possible extension in which CC-AlignBench evaluations are additionally organized by standardized human-preference axes, with coverage metrics such as

$I_g = \theta(T, I)$6

and

$I_g = \theta(T, I)$7

or preference-specific leaderboards such as pElo and category-weighted optimization objectives such as weighted DPO (Vodrahalli et al., 31 Mar 2025).

Taken together, these two lines of work situate CC-AlignBench within a broader movement toward decomposed, interpretable, and explicitly human-aligned evaluation. In its current form, CC-AlignBench operationalizes that goal through staged human-centric prompts and 18 aspect-wise MLLM judgments. A plausible implication is that future versions could couple this image-centric decomposition with preference-basis diagnostics that expose which human-valued properties are being preserved, missed, or traded off across concept customization systems (Ishikawa et al., 3 Sep 2025).