GenEval++: Advanced Text-to-Image Evaluation

• GenEval++ is an extended evaluation framework that offers object-focused, compositional, and fine-grained assessments of text-to-image alignment in generative models.
• It integrates pre-trained segmentation and zero-shot classification models to quantitatively assess object presence, count, spatial relationships, and color accuracy from text prompts.
• Advancements such as open-vocabulary detection, continuous scoring, and adversarial testing address key bottlenecks and improve evaluation robustness for next-generation generative systems.

GenEval++ is a conceptual extension of the GenEval framework, designed to provide object-focused, compositional, and fine-grained automated evaluation of text-to-image alignment in generative models. GenEval employs pre-trained object detection and segmentation models alongside discriminative vision-LLMs to quantitatively assess whether images generated from text prompts satisfy concrete compositional constraints such as object co-occurrence, position, count, and color. GenEval++ expands these capabilities by proposing open-vocabulary detection, extended attribute evaluation, continuous scoring, and adversarial testing, thus addressing both current evaluation bottlenecks and the increasingly complex demands of next-generation text-to-image generation models (Ghosh et al., 2023).

1. Formal Definitions and Core Evaluation Criteria

Let $I$ denote a generated image and $P$ the associated text prompt. Prompts are parsed into sets of objects and potentially precise constraints: object presence, required counts $n_i$, colors $c_i$, and spatial relations $R_{ij}$. GenEval leverages a pre-trained detector $D$ to extract detections:

$D(I) = \{(o, b, s, m)\}$

where $o$ is the detected object class, $b$ the bounding box, $s \in [0, 1]$ a confidence score, and $m$ the instance mask.

The evaluation predicates are:

• Object Presence: $Presence(o,I)$ holds if $\exists (o', b, s, m) \in D(I)$ such that $o' = o$ and $s \geq \tau_{det}$ with typical $\tau_{det} = 0.3$. The image satisfies the co-occurrence criterion if all demanded objects are present.
• Object Counting: $$Count(o, I) = |\{(o', b, s, m) \in D(I) \mid o' = o, s \geq \tau_{count}\}|$$ with $\tau_{count} = 0.9$. The counted number must match $n$ indicated in the prompt.
• Spatial Relationships: For detected instances $A$, $B$, with centroids $c_A = (x_A, y_A)$, $c_B = (x_B, y_B)$, and box sizes $w_A, h_A, w_B, h_B$, spatial predicates use a margin $c = 0.1$:
  • $$Right(A, B; I) \Longleftrightarrow x_B - x_A > c \cdot (w_A + w_B)$$,
  • $Left(A, B; I)$, $Above(A, B; I)$, $Below(A, B; I)$ are defined analogously.
  • The spatial predicate must be satisfied for the pair of instances with highest confidence.
• Color Accuracy: For $(o, c)$, the top-confidence detection is cropped and masked. The masked crop is classified by a zero-shot CLIP model over a fixed set of colors $C$ using cosine similarity between image and text embeddings. Prediction is correct iff the top-1 predicted color matches the prompt's color designation.

All prompt-constrained predicates must pass for an image to be counted as correct.

2. Pipeline Architecture

GenEval is a modular pipeline constructed atop two primary models:

• Instance Segmentation: Mask2Former from MMDetection provides object detection and segmentation masks.
• Color Classification: The CLIP ViT-L/14 model is employed for zero-shot color classification.

The pipeline proceeds as follows:

• Parse the prompt $P$ into object, count, relation, and color constraints.
• Run Mask2Former on $I$ to obtain detections.
• Validate each constraint:
  • Presence: Require at least one detection per object class above threshold.
  • Counting: Exact required count per class with a higher threshold to avoid duplicates.
  • Spatial: Evaluate the relevant predicate using bounding box centroids and sizes.
  • Color: Crop and mask the detected object, classify against candidate colors using CLIP.

Correctness is a binary predicate: an image passes if all constraints are satisfied. Scores are aggregated as averages over images to form task-level and model-level metrics.

3. Metrics, Thresholds, and Performance Computation

GenEval employs task-specific thresholds and metrics:

• Thresholds: $\tau_{det} = 0.3$ for most tasks; $\tau_{count} = 0.9$ for counting.
• Color: Top-1 prediction over 10 Berlin–Kay colors with backgrounds grayed out.
• Binary Metric: All constraints on presence, count, relation, and color must be satisfied for a "correct" verdict.
• Task Score: Proportion of correct images for each constraint type.
• Overall Score: Mean over the six primary task scores.
• Human Alignment: Evaluated using agreement rates and Cohen's $\kappa$; GenEval achieves $\sim$83% agreement ($\kappa \approx 0.88$), superior to CLIPScore particularly for compositional tasks.
• CLIPScore Baseline: CLIP embedding cosine similarity (thresholded) performs only on simple presence, not well for counting, spatial, or attribute binding tasks.

4. Empirical Assessment and Comparative Results

GenEval systematically evaluates a suite of open-source text-to-image models. The main findings are summarized in the following table (from Table 3 of (Ghosh et al., 2023)):

Model	Single	Two-Object	Counting	Color	Position	Attr-Bind	Overall
CLIP-retrieval	0.89	0.22	0.37	0.62	0.03	0.00	0.35
minDALL-E	0.73	0.11	0.12	0.37	0.02	0.01	0.23
SD v1.5	0.97	0.38	0.35	0.76	0.04	0.06	0.43
SD v2.1	0.98	0.51	0.44	0.85	0.07	0.17	0.50
SD XL 1.0	0.98	0.74	0.39	0.85	0.15	0.23	0.55
IF-XL	0.97	0.74	0.66	0.81	0.13	0.35	0.61

Key findings:

• State-of-the-art diffusion models achieve near-ceiling performance on single-object presence ($>$97%) and color ($\sim$80–85%).
• Marked improvements in two-object co-occurrence for IF-XL and SD-XL ($\sim$74%), compared to older models.
• Counting remains challenging, with best observed performance at 66% (IF-XL).
• Spatial relations performance is poor ($\leq 15\%$), with little gain from scaling.
• Attribute binding tasks (distinct color assignment across objects) remain a bottleneck ($\leq 35\%$ for IF-XL).
• Scaling up model size improves co-occurrence, counting, and attribute binding, but not spatial relation reasoning, indicating domain-specific limitations.

5. Identified Failure Modes and Mitigation Strategies

Failure modes are observed on both discriminative and generative outputs:

• Discriminative Model Failures:
  • Mask2Former mis-segments objects with complex topologies (e.g., internal holes).
  • Highly overlapping instances hinder accurate counting.
  • COCO-trained detectors generalize poorly to stylized or non-photorealistic images.

Proposed solution involves replacing standard detectors with open-vocabulary instance segmentation models (e.g., OWL-ViT, Grounding DINO) trained on diverse, unconstrained datasets to handle out-of-distribution (OOD) samples more robustly.

• Generative Model Failures:
  • Persistent spatial bias, e.g., models generating "A above B" often default to lateral rather than vertical layouts regardless of prompt sampling.
  • Attribute binding errors, manifesting as color swaps or color leakage across objects or backgrounds.
  • Failure on complex scenes with multiple objects and nested relationships.

Proposed generative-side strategies, based on related work, include synthetic fine-tuning on explicitly compositional 3D datasets, auxiliary spatial-relation objectives, and reinforcement signals targeting correct object arrangement.

6. Extensions: GenEval++ Directions

GenEval++ outlines several next-generation directions:

• Open-Vocabulary and Fine-Grained Semantics: Adoption of segmentation models (e.g., Grounding DINO, OWL-ViT) to address arbitrary object categories and attributes beyond the COCO taxonomy.
• Extended Attribute Classification: Zero-shot classifiers for attributes such as texture ("furry", "shiny"), material ("wooden", "metal"), and dynamic actions ("jumping", "holding"), extending beyond the current color corpus.
• Hierarchical and Relational Structure: Scene-graph reconstruction capabilities, enabling evaluation over sets of $(\langle subject, predicate, object \rangle)$ triplets, possibly via joint VQA or graph parsing modules.
• Incorporation of Vision–LLMs (VLMs): For free-form or ambiguous prompts, use LLM-based parsing to create atomic evaluation tasks, with VQA models (BLIP-2, Flamingo) serving as a fallback when traditional detection pipelines are insufficient.
• Continuous and Calibrated Scoring: Shift from binary metrics to confidence-weighted or calibrated scores for each predicate, aggregated using learned or human-aligned weighting schemes.
• Benchmark Expansion: New tasks targeting occlusion, viewpoint, style transfer, and fine-grained part recognition.
• Adversarial and Stress Testing: Automatic paraphrasing and targeted prompt modifications to probe and document generative model failure modes.

These proposed advances aim to systematically extend the scope and precision of compositionality and attribute-grounded evaluation for text-to-image generative systems.

7. Context and Significance

GenEval demonstrates that pipeline-style, modular evaluation leveraging object segmentation and discriminative vision-LLMs can yield interpretable, high-fidelity evaluation of compositional text-to-image generation, closely aligned to human judgment. Adoption of such frameworks is essential given the limits of holistic metrics (e.g., FID, CLIPScore) for compositional, instance-level, or relational correctness. GenEval++—as outlined—responds to current challenges in both evaluation robustness and model capabilities, indicating a research direction grounded in open-vocabulary detection, richer attribute classification, hierarchical scene understanding, and nuanced, human-aligned metrics (Ghosh et al., 2023).