TXIT Performance Evaluation

• The paper introduces a novel evaluation framework for TTI systems that uses prompt-specific, set-level metrics grounded in models of user browsing behavior.
• It adapts IR metrics like Rank-Biased Precision and Expected Reciprocal Rank to quantify fluency, quality, variety, and novelty while penalizing visual redundancy.
• Empirical validation with human studies shows that these metrics outperform traditional global measures like FID in capturing creative ideation support.

TXIT performance evaluation refers to the systematic assessment of performance for Text-to-Image Ideation Tasks (TTI), where the core goal is to benchmark how effectively a TTI system supports creative ideation via its generated image sets. Recent work has introduced offline evaluation metrics that are grounded in models of human browsing behavior and diverge from traditional population-level metrics such as Fréchet Inception Distance (FID). These new metrics model user interaction, fluency, quality, diversity, and arrangement, offering refined criteria for gauging system efficacy for ideation-driven workflows (Arabzadeh et al., 2024).

1. Evaluation Metrics for Set-Based TTI

The proposed evaluation framework departs from metrics like FID that summarize global distributional similarity, focusing instead on prompt-specific, set-level assessment. The evaluation hinges on four core criteria:

• Fluency: total count of relevant images per set.
• Quality: prompt-relevance of each image in the set.
• Variety: degree to which images represent unique ideas.
• Novelty: distinctiveness of each image compared to images already examined.

Metrics are derived from information retrieval (IR), specifically Rank-Biased Precision (RBP) and Expected Reciprocal Rank (ERR), and are adapted to model user browsing trajectories over image grids. These formulations include discount factors penalizing visual redundancy:

$$RBP(\tau) = \sum_{i=1}^k f^*(x_{\tau_i}) \cdot \gamma^{i-1}$$

$$ERR(\tau) = \sum_{i=1}^k f^*(x_{\tau_i}) \cdot \prod_{j=1}^{i-1} [1 - g^*(x_{\tau_j})]$$

$$d(i) = 1 - \max_{j \in [1,i-1]} \text{Similarity}(x_{\tau_i}, x_{\tau_j})$$

Here, $f^*(x)$ is estimated image relevance, $\gamma$ is the inspection persistence parameter, and $\tau$ is the grid traversal order. The discount function $d(i)$ reduces the metric’s value for visually repetitive content.

2. Modeling User Interaction and Creative Ideation

The evaluation is explicitly motivated by real-world user interaction during creative ideation, when users scan grids for relevant and diverse visual concepts. Two browsing models are implemented:

• Position-based Model: supports exponentially decaying probability of inspecting deeper grid positions ($\gamma^{i-1}$).
• Cascade Model: incorporates the probability that a sufficiently relevant image prompts the user to stop exploring ($g^*(x)$).

These models simulate realistic creative tasks—such as compiling a mood board or selecting design directions—where the process benefits from both quantity (fluency) and variety (novelty).

3. Diversity, Spatial Arrangement, and Salience

A primary innovation is explicit modeling of diversity and grid arrangement. Similarity penalties within the metric encourage presentation of images spanning different areas of the design space. User trajectories are probabilistically sampled (via Plackett-Luce distributions and techniques like Gumbel-softmax) according to predicted visual salience and spatial layout, acknowledging non-sequential inspection patterns.

This approach contrasts with naive list-based ranking, and accounts for the fact that image grids (two-dimensional arrays) affect which images attract user attention. Figures in the source material illustrate this via annotated grids and saliency maps.

4. Empirical Validation with Human Studies

Human evaluation experiments were conducted on image grids from three TTI systems and datasets (MS-COCO captions, Localized Narratives, and naturalistic prompts). Annotators used Likert scales to judge sets in simulated ideation tasks. Metrics based on sequential user models (e.g., ERR with novelty discounts, RBP variants) exhibited higher concordance with human judgments, particularly for complex prompts requiring nuanced creative exploration.

Statistical tests (Wilcoxon paired, $p < 0.05$) confirmed the superiority of these metrics in differentiating system performance. Simpler diversity metrics (average pairwise similarity) did not correlate well with perceived creative support.

5. Benchmark Design Principles

The paper advocates for evaluation protocols that reflect the interactive nature of TTI use in creative workflows. Benchmarks should:

• Incorporate behavioral models simulating browsing of spatial image sets.
• Measure fluency, quality, variety, and novelty per prompt.
• Include trajectory sampling and redundancy penalties.
• Move beyond one-size-fits-all metrics like FID in favor of prompt- and user-centric evaluation.

This approach improves both validity and interpretability of performance assessments, aligning metric design with actual user needs during ideation.

6. Significance and Outlook

Grounding TTI evaluation in models of user interaction and spatial grid browsing offers substantial improvements over global statistical metrics. The combination of adapted IR metrics and human studies establishes a foundation for benchmark methods more sensitive to creative tasks. Future TTI benchmarks will likely further integrate interactive, task-driven measures, guiding system development toward practical utility in design and ideation-oriented domains (Arabzadeh et al., 2024).