Generative Evaluation Protocol

• Generative Evaluation Protocol is a formal workflow that quantifies attribute-level strengths in generative models using metrics like Heterogeneous CLIPScore (HCS).
• It employs divergence measures (SaD and PaD) via density estimation to diagnose deviations in attribute representation and joint attribute relationships.
• The protocol provides actionable insights by enabling detailed diagnostic drilldowns, thereby complementing traditional global metrics in model assessment.

A generative evaluation protocol is a formal, reproducible workflow for measuring the quality, fidelity, and diagnostic characteristics of generative models—whether for text, images, or other modalities. Recent advances emphasize interpretable, attribute-level analysis, offering actionable insights into which features of the target distribution a generative model reproduces faithfully or distorts. The protocol detailed in "Attribute Based Interpretable Evaluation Metrics for Generative Models" (Kim et al., 2023) is predicated on a workflow that centers around attribute-strength estimation via Heterogeneous CLIPScore (HCS), divergence-based metrics (Single-attribute Divergence [SaD] and Paired-attribute Divergence [PaD]), and granular diagnostic reporting.

1. Principles of Attribute-Centric Evaluation

The protocol asserts that evaluation should not solely quantify pixel- or embedding-level similarity, but instead parse and compare the distribution of interpretable attributes in the generated outputs relative to the reference data. It operationalizes this via:

• A user-specified attribute vocabulary $\mathcal{A}$, tailored to the domain (e.g., "beard," "long hair").
• Scalar "attribute strengths" per image–attribute pair, which are derived as a centered CLIP image–text cosine similarity, termed Heterogeneous CLIPScore (HCS).

This methodology enables both isolating attribute-specific overand undergeneration and detecting implausible joint attribute combinations, which would be invisible to standard metrics such as FID or Inception Score.

2. Heterogeneous CLIPScore (HCS): Scalar Attribute Strengths

For any real image set $\mathcal{X} = \{ x_1, \ldots, x_{N_x} \}$ and attribute set $\mathcal{A} = \{ a_1, \ldots, a_{N_a} \}$:

• Compute the image and text "centers" in CLIP embedding space:

$$C_\mathcal{X} = \frac{1}{N_x}\sum_{i=1}^{N_x} E_I(x_i), \quad C_\mathcal{A} = \frac{1}{N_a}\sum_{j=1}^{N_a} E_T(a_j)$$

• For any image $x$ and attribute $a$, generate centered vectors:

$$V_x = E_I(x) - C_\mathcal{X}, \quad V_a = E_T(a) - C_\mathcal{A}$$

• Define the Heterogeneous CLIPScore:

$$HCS(x, a) = 100 \cdot \mathrm{cosine\_similarity}(V_x, V_a)$$

HCS is calculated across all training images and all generated images, for each attribute. This scalar is then used to map the empirical distribution of attribute strengths.

3. Attribute Distribution Divergence Metrics

After attribute strengths are extracted, the protocol fits continuous probability distributions (PDFs) to both the real and generated sets:

• For each attribute $a_i$, fit 1-d PDFs $f_\mathcal{X}(a_i)(t)$ and $f_\mathcal{Y}(a_i)(t)$ via Gaussian kernel density estimation (KDE), following Scott's rule.
• For each attribute pair $(a_i, a_j)$, fit joint 2-d PDFs.

The two core divergence metrics are:

• Single-attribute Divergence (SaD):

$$SaD(\mathcal{X}, \mathcal{Y}) = \frac{1}{M} \sum_{i=1}^M KL(f_\mathcal{X}(a_i) \| f_\mathcal{Y}(a_i))$$

Here $$KL(p \| q) = \int p(t) \log \left[ \frac{p(t)}{q(t)} \right] dt$$ quantifies how well each attribute's strength distribution is reproduced.

• Paired-attribute Divergence (PaD):

$$PaD(\mathcal{X}, \mathcal{Y}) = \frac{1}{|P|} \sum_{(i,j) \in P} KL(f_\mathcal{X}(a_i, a_j) \| f_\mathcal{Y}(a_i, a_j))$$

This metric exposes joint attribute relationships—revealing implausible combinations such as "baby ∧ beard."

Large SaD or PaD values indicate systematic deviations in either marginal attribute representation or cross-attribute conjunctions.

4. Step-by-Step Evaluation Workflow

The canonical pipeline proceeds as follows:

1. Attribute Selection: Define $\mathcal{A}$, either by extracting frequent words from captions (e.g., with BLIP), or via curated annotations.
2. Strength Computation: Calculate HCS(x,a) for every training image $x \in \mathcal{X}$ and attribute $a \in \mathcal{A}$.
3. Generation Analysis: Compute HCS(y,a) for large batches of generated images $y \in \mathcal{Y}$ (10,000–50,000 samples typical).
4. Density Estimation: For each attribute and attribute pair, fit PDFs (1D and 2D, respectively) to the empirical HCS values.
5. Divergence Scoring: Compute SaD and PaD by KL divergence between corresponding PDFs.
6. Diagnostic Drilldown: Optionally, inspect per-attribute and per-pair KL contributions to diagnose most misaligned attributes or combinations.

5. Empirical Diagnostics and Applications

Several findings illuminate the protocol's diagnostic resolution:

• Implausible Attribute Combinations (e.g., ProjectedGAN): PaD identifies "baby with beard" co-occurrences in ProjectedGAN that are competitive by naive metrics but semantically impossible.
• Marginal Attribute Gaps (Diffusion Models): SaD reveals that diffusion-based models (iDDPM) systematically fail to synthesize diverse color patterns, with SaD for color attributes in iDDPM one-third that of StyleGAN.
• Sampling Timesteps and Minor Object Overgeneration: More sampling steps in Latent Diffusion Models yield FID improvements but worsen SaD/PaD, and lead to over-generation of minor attributes (e.g., jewelry).
• Model Variant Comparison (Stable Diffusion v1.5 vs v2.1): SaD and PaD evidence that v1.5 better preserves object attributes. v2.1's higher SaD/PaD co-occur with missing objects such as "group," "plate," or "person."

This fine-grained reporting allows detailed root-cause analysis and guides model retraining and tuning for specific distributional fidelity.

6. Relation to Prior Metrics and Interpretability

Traditional metrics—Fréchet Inception Distance (FID), Inception Score (IS), precision/recall, density/coverage—summarize global distributional overlap, but lack semantic interpretability. They cannot attribute errors to specific attributes or pairs, thereby obscuring actionable model failures. In contrast, the SaD/PaD metrics paired with HCS decompose divergence along meaningful axes, directly enabling model debugging and targeted correction.

• Explainability: Each divergence is traceable to specific attribute under-/over-representation or implausible combinations.
• Actionability: Identification of specific gaps means model architectures or postprocessing can be modified to correct those attributes.

This attributewise transparency situates the protocol as an explicitly interpretable alternative to prior black-box metrics.

7. Protocol Limitations and Future Refinements

The protocol, while interpretable and actionable, is bounded by:

• The coverage and granularity of the attribute vocabulary $\mathcal{A}$. Under-specified attributes will lead to incomplete diagnostics.
• The quality of the underlying CLIP encodings and their centering, which may be vulnerable to domain- or dataset-specific embedding biases.
• Kernel density estimation's sensitivity to sample count and bandwidth (Scott’s rule), which may misestimate attribute PDFs for low-frequency attributes.
• Assumption of independence between attribute measurements, unless explicitly modeled in higher-order joint divergence (beyond paired).

Future protocol revisions may incorporate multi-attribute or hierarchical joint divergences, improved text-image embedding models, adaptive kernel estimation, and domain-specific attribute sets to broaden diagnostic resolution.

---

In summary, the attribute-based generative evaluation protocol (Kim et al., 2023) delivers interpretable, feature-level fidelity diagnostics for generative models, superseding global-only metrics and providing mechanisms for precise, actionable model improvement. Its methodology—HCS for scalar attribute measurement, SaD/PaD for distributional divergence, and granular logging—makes it a foundational tool for rigorous generative model assessment and debugging.