3SGen-Bench: Multimodal Image Generation Benchmark

• 3SGen-Bench is a unified benchmark that evaluates image generation models by isolating subject, style, and structure conditioning modes.
• It uses standardized datasets and metrics, including CLIP-I, DINO, and VLM-based scores, to rigorously compare multimodal generative architectures.
• Results demonstrate that 3SGen outperforms baselines in fidelity and controllability, effectively addressing feature entanglement and cross-mode interference.

3SGen-Bench is a unified benchmark for image-driven generation that systematically evaluates models across subject, style, and structure conditioning modes. Developed within the context of the 3SGen framework, the benchmark isolates and quantifies the fidelity and controllability of generated images with respect to multiple modalities of input and compositional settings. Its standardized datasets, metrics, and scoring apparatus enable rigorous, like-for-like comparison of diverse multimodal generative architectures and disentanglement mechanisms (Song et al., 22 Dec 2025).

1. Benchmark Scope and Task Taxonomy

3SGen-Bench is constructed to reflect the three principal conditioning modes in contemporary image generation:

1. Subject-driven generation: The model receives at least one reference image depicting a specific subject—such as a person, animal, or object—paired with a text prompt. The goal is to synthesize novel images retaining the subject’s identity while incorporating prompted semantics.

1. Style-driven generation: Given a style reference (e.g., a painting, photograph, or digital artwork) and an input prompt, the model produces images adopting the global color palette and texture cues of the reference style, adapting these to the prompted content.
2. Structure-driven generation: The model is guided by a reference structural map (Canny edge, depth map, HED contours, sketches, pose keypoints, or surface normals) plus prompt text. The output is evaluated for fidelity to the input spatial layout as well as semantic relevance to the prompt.

For each mode, 100 held-out prompts are constructed. The subject set spans diverse semantic categories, style prompts cover 40 distinct types, and structure tasks include six input modalities. Four stochastic samples are generated per prompt, resulting in 400 output images per conditioning mode, for a total of 1,200 single-source test images.

Compositional tasks extend the benchmark to multi-condition input, including “subject+style” and “style+structure”; however, while 50,000 combinatorial training triplets are used during model development, the compositional evaluation split comprises all possible pairs among the 100 subject and 100 style (or structure) prompts, also with four generated images per prompt pair.

2. Evaluation Metrics

Quantitative evaluation in 3SGen-Bench targets both conditioning-source fidelity and semantic controllability with prompt adherence. Metrics are divided into off-the-shelf embedding measures and VLM-based learned scores:

a. Embedding-based Metrics

• CLIP-I: Subject-consistency measured by cosine similarity between CLIP image embeddings of generated and reference subject images:

$$\text{CLIP-I}(I_{\text{gen}}, I_{\text{ref}}) = \frac{\langle E_\text{clip-img}(I_{\text{gen}}), E_\text{clip-img}(I_{\text{ref}}) \rangle}{\|E_\text{clip-img}(I_{\text{gen}})\| \cdot \|E_\text{clip-img}(I_{\text{ref}})\|}$$

• DINO: Alternative subject-consistency using DINOv2 embeddings.
• CSD: Style-consistency via cosine similarity of Gram-matrix style descriptors:

$$\text{CSD}(G_{\text{gen}}, G_{\text{ref}}) = \frac{\langle G_{\text{gen}}, G_{\text{ref}}\rangle}{\|G_{\text{gen}}\| \cdot \|G_{\text{ref}}\|}$$

• Struc-Sim: Structure-consistency, evaluated as normalized $\ell_1$ matching between structural maps:

$$\text{Struc-Sim}(S_{\text{gen}}, S_{\text{ref}}) = 1 - \frac{1}{WH}\sum_{i,j} |S_{\text{gen}}(i,j) - S_{\text{ref}}(i,j)|$$

• CLIP-T: Prompt fidelity by comparing CLIP embeddings between generated images and text prompts.
• FID: Fréchet Inception Distance quantifying visual realism relative to real images from the underlying data distribution.

b. Learned VLM-Based Scores

Utilizing the Qwen2.5-VL vision-LLM, the benchmark assigns four scalar “conditional consistency” scores per output in the range $[1, 10]$:

Criteria	Description
Subject Consistency	Identity match between generated and reference subject
Style Consistency	Faithful style transfer relative to reference style
Structure Consistency	Adherence to spatial layout of input structure
Prompt Fidelity	Alignment with the semantic intent of the prompt

Scores are averaged over four samples and all 100 prompts per task. The internal mechanism for Qwen2.5-VL scoring remains proprietary.

3. Protocol and Implementation

Evaluation protocol is strictly controlled for reproducibility:

• Three isolated subtasks (subject, style, structure) are each tested on 100 held-out prompts and references.
• Models generate four outputs per prompt using matched sampler configurations.
• Metrics are computed directly on these outputs relative to their respective reference images, style descriptors, or structure maps.
• Conditional consistency scores (via Qwen2.5-VL) and human expert assessment (1–10 scale on four axes) form the core of reported results.
• Manual review by expert panel is conducted for diagnostic user studies, supplementing automated quantitative results.

Baseline methods for direct comparison include:

Task	Baselines
Subject/Style	OmniGen2, FLUX Kontext dev, Qwen-Image Edit, USO, UnicAdapter
Structure	ControlNet, UniControl, UnicAdapter
Composition	USO (subject+style); UnicAdapter + USO cascade (style+structure)

All baselines receive identical inputs and sampler settings, ensuring fairness.

4. Key Results and Comparative Analysis

Benchmark findings consistently indicate that 3SGen achieves superior performance across all axes, with conditional consistency scores surpassing all unified and task-specific baselines:

• By Table 2 in the source, 3SGen attains subject (8.41), style (7.35), structure (8.22), and prompt (8.67) scores—highest in their respective columns.
• Subject-driven generation: 3SGen achieves optimal tradeoff between identity preservation and sample diversity, while comparators such as UnicAdapter and FLUX “copy-paste” reference content, inflating CLIP-I at the expense of semantic fidelity and naturalistic realism.
• Style-driven: Methods lacking effective disentanglement (OmniGen2, Qwen-Image Edit) result in leakage of subject features into styled output; ATM in 3SGen cleanly isolates style vectors yielding improvements in CSD and CLIP-T.
• Structure-driven: Joint MLLM-VAE encoding enables high-fidelity layout adherence, reflected in approximately double Struc-Sim versus ControlNet and UniControl.
• Compositional inputs: Baselines suffer from unbalanced conditioning (e.g., style dominance erasing structure cues), whereas adaptive gating in 3SGen selectively retrieves condition-specific priors, preserving both style and spatial layout.

Identified failure modes in baseline methods include inter-condition entanglement, diminished prompt adherence under conflicting conditions, and excessive subject copying leading to low diversity.

5. Benchmark Adoption and Significance

3SGen-Bench is open-sourced for public adoption, affording researchers reproducible data splits, standardized prompts, reference sets, and evaluation pipelines for comparative multimodal generative model analysis. Its unified framework for subject, style, and structure conditioning — alongside robust automated and manual scoring — addresses longstanding challenges of feature entanglement and cross-mode interference prevalent in prior benchmarks.

A plausible implication is that widespread benchmarking on 3SGen-Bench will facilitate development of models capable of more precise, disentangled, and compositional image synthesis, as well as enable deeper scrutiny into the interaction between conditioning signals and prompt semantics.

6. Relation to Other Benchmarks

While 3SGen-Bench is focused on 2D image generation with compositional multimodal conditioning, benchmarking efforts in adjacent modalities—such as 3DGen-Bench for 3D generative models (Zhang et al., 27 Mar 2025)—adopt human preference annotation and automated evaluators for model ranking. The comparative emergence of CLIP and MLLM-based scoring functions in both domains suggests a trend toward embedding-based and VLM metric standardization; nonetheless, 3SGen-Bench is distinguished by its fine-grained conditioning split and compositional evaluation schema, which serve as a template for future multimodal image synthesis benchmarks.

Researchers analyzing cross-task fidelity and developing adaptive or disentangled architectures for image generation may reference 3SGen-Bench as a canonical evaluation protocol, leveraging its reproducible methodology for both single-source and compositional conditioning setups.