Magic Evaluation Framework (MEF) for T2I Models

• MEF is a comprehensive evaluation framework for T2I models that uses a multi-level taxonomy and Magic-Bench-377 to capture both application scenarios and objective capabilities.
• It integrates ELO ranking with MOS scoring to provide overall competitive standings alongside detailed assessments of prompt following, structural accuracy, and aesthetic quality.
• The framework employs multivariate logistic regression to quantify contributions of each evaluation dimension, enabling nuanced insights into model performance across diverse user groups.

The Magic Evaluation Framework (MEF) is a systematic and practical framework for evaluating text-to-image (T2I) models. It was introduced to address two deficiencies in prior T2I evaluation practice: existing benchmarks often emphasize objective capabilities without an application-scenario perspective, and prevailing evaluation methods typically rely on either ELO for overall ranking or MOS for dimension-specific scoring, although both have inherent shortcomings and limited interpretability. MEF therefore combines a structured taxonomy, a benchmark called Magic-Bench-377, a hybrid evaluation protocol based on ELO and MOS, and an end-to-end quality management system to support both leaderboard construction and fine-grained diagnostic analysis (Dong et al., 22 Sep 2025).

1. Conceptual scope and design goals

MEF is organized around three core components: a multi-level structured taxonomy guiding benchmark construction, a hybrid evaluation protocol combining ELO ranking and multidimensional MOS scoring, and an end-to-end quality management system (Dong et al., 22 Sep 2025). Its central objective is not merely to order models by aggregate preference, but to make T2I evaluation more application-aware, label-aware, and diagnostically interpretable.

The framework distinguishes between real-world user scenarios and objective capabilities. That distinction is foundational. Scenario-aware evaluation addresses external validity by asking how models behave in use contexts such as film, art, entertainment, aesthetic design, and functional design. Capability-aware evaluation, by contrast, isolates what a model must do semantically and compositionally. MEF treats these as complementary rather than competing views of model quality.

A further design goal is methodological complementarity. ELO provides relative competitive ranking, while MOS provides absolute, dimension-specific assessment. MEF joins them rather than substituting one for the other. The framework then uses multivariate logistic regression to relate dimension-level MOS signals to ELO preferences, thereby quantifying how much each evaluation dimension contributes to user satisfaction.

2. Taxonomy and benchmark construction

MEF introduces a multi-level taxonomy with two primary branches: Application Scenarios and Objective Capabilities (Dong et al., 22 Sep 2025). The application-scenario branch contains five categories: Film, Art, Entertainment, Aesthetic Design, and Functional Design. The objective-capability branch is divided into three major categories: Element, Element Composition, and Text Expression Form.

Within Element, the framework includes Entity, Entity Description, and Image Description. Within Element Composition, it includes Multi-Entity Feature Matching, Layout/Typographic Composition, and Anti-Realism. Within Text Expression Form, it includes Negation, Pronoun Reference, and Consistency. This taxonomy is meant to support precise prompt annotation and label-level performance analysis rather than coarse benchmark totals.

Branch	Internal structure	Representative items
Application Scenarios	Five scenario categories	Film, Art, Entertainment, Aesthetic Design, Functional Design
Objective Capabilities	Three major categories	Element, Element Composition, Text Expression Form
Capability subtypes	Finer-grained labels	Entity, Layout/Typographic Composition, Negation

The benchmark built from this taxonomy is Magic-Bench-377, a set of 377 prompts with detailed multi-label annotations. Each prompt is tagged with 1–4 objective capability labels and one application scenario. The prompts are intentionally designed to combine multiple test points so that evaluation reflects the cross-capability interference of real usage rather than only atomized skills. Prompt creation involved professional designers, AI tool power users, and annotation experts, and a multi-phase quality inspection process was used to ensure prompt clarity, fairness, label diversity, and neutrality, including freedom from copyrighted or culturally specific elements.

This benchmark design yields three stated advantages. First, it supports label-level assessment, because outputs can be analyzed by capability tag rather than only by overall score. Second, it maintains balanced capability and scenario coverage. Third, its hierarchical organization provides a basis for extensibility, since new test points can be added under the existing taxonomy.

3. Hybrid evaluation protocol

The MEF evaluation protocol combines ELO and MOS so that comparative ranking and dimension-specific diagnosis are produced within the same framework (Dong et al., 22 Sep 2025). In the ELO component, model outputs are presented pairwise, side-by-side, with randomization and anonymization. Annotators choose among left wins, right wins, both good, or both bad. These pairwise judgments are based on overall quality, which integrates Prompt Following, Structural Accuracy, and Aesthetic Quality.

Leaderboard estimation uses a Bradley-Terry model in the style of Chatbot Arena. The preference probability is written as

$P(H_t = 1) = \frac{1}{1 + e^{\xi'_{m} - \xi_m}}.$

MEF reports ELO scores overall, by scenario, and per prompt, and confidence intervals are computed by bootstrap resampling. A minimum participation threshold of at least 4000 matches per model is imposed.

The MOS component assigns 1–5 scores on three decoupled dimensions: Prompt Following, Structural Accuracy, and Aesthetic Quality. Prompt Following measures faithfulness to the prompt semantics. Structural Accuracy measures completeness, coherence, and plausibility of entity arrangement. Aesthetic Quality measures visual and artistic appeal, including style, color, and composition. To reduce stochastic variation, each model generates 4 images per prompt, and experts score each of them; the reported effect is that variance drops from 30% to less than 5%.

MEF explicitly attempts to decouple the three MOS dimensions. Raters are trained to focus on a single dimension at a time and use clear deduction rules to minimize dimension coupling. The reported empirical evidence is a Pearson correlation $r < 0.3$ between dimensions. Statistical practice also receives explicit treatment: confidence intervals account for prompt-level clustered sampling, and MOS differences larger than 0.1 are treated as statistically significant.

4. Quality management and statistical interpretation

MEF includes a quality-management layer intended to stabilize human evaluation and reduce annotation drift (Dong et al., 22 Sep 2025). The framework specifies expert raters, qualification, consistency checks, and periodic re-certification. In addition, it uses 5% anchor items and 25% audits as quality-control sampling procedures. The ELO process is also described as double-blind, with random sampling and anti-cheating mechanisms that include anchor items and temporal behavior analysis.

A distinctive feature of MEF is its use of multivariate logistic regression to quantify how MOS dimensions drive ELO outcomes. With standardized MOS features, the model is written as

$$\log\left(\frac{p}{1-p}\right) = \beta_1 \cdot x_1 + \beta_2 \cdot x_2 + \beta_3 \cdot x_3 + \text{(interactions)}.$$

Here, $x_1, x_2, x_3$ correspond to standardized scores for Prompt Following, Structural Accuracy, and Aesthetic Quality. Interaction terms are included to capture non-linear effects.

The reported interpretation is that Prompt Following is always the most influential dimension for user preference, especially among experts and in all scenarios except Art, where aesthetics can partially offset prompt mismatch. The persona-specific contribution analysis is particularly salient. For users, the contributions are 12.5% for Prompt Following, 0.7% for Structural Accuracy, and 7.1% for Aesthetic Quality. For experts, the contributions are 37.7%, 14.2%, and 10.9%, respectively. For designers, they are 26.0%, 10.0%, and 22.1%. These values operationalize differences in evaluative preference across populations rather than assuming a single universal utility function.

5. Benchmark results and model differentiation

MEF was applied to a set of contemporary T2I systems including GPT-4o, Seedream 3.0, Imagen 3, Ideogram 3.0, and Midjourney V6.1 (Dong et al., 22 Sep 2025). The framework reports both expert and public ELO leaderboards and supplements them with scenario-specific and label-level analyses.

Model	ELO (expert/public)	Characteristic strength
GPT-4o	1205 / 1067	Semantic and structural mastery
Seedream 3.0	1178 / 1084	Balanced performance and best aesthetic
Imagen 3	1100 / 1031	Balanced
Ideogram 3.0	1078 / 1007	Text rendering specialist
Midjourney V6.1	1000 / 989	Top artistic aesthetics

The aggregate rankings do not imply uniform superiority. MEF emphasizes that no model is uniformly superior. Scenario-specific ELO and MOS analyses show, for example, that Midjourney V6.1 rises in Art scenarios because aesthetic quality carries greater weight there. Capability-level analysis indicates that GPT-4o leads in tasks requiring reading text, logical composition, or negation, while Seedream 3.0 excels at textures and physical interaction. At the same time, the label-level breakdown reveals recurring difficulty on comparative relations and intricate compositional tasks.

These results illustrate the framework’s central methodological claim: leaderboard position alone is too coarse to characterize model behavior. MEF instead treats model quality as a distribution over scenarios, capabilities, and evaluator populations.

6. Methodological significance and interpretive issues

MEF positions itself as an application-aware alternative to evaluations that either isolate objective capabilities too narrowly or reduce model quality to a single ranking statistic (Dong et al., 22 Sep 2025). Because Magic-Bench-377 combines multiple test points within prompts, the benchmark is intended to mirror real user prompts rather than idealized single-skill probes. The framework therefore aims to avoid overestimating actual usability.

The combination of relative and absolute evaluation is also methodologically significant. ELO provides a competitive ranking that is easy to communicate, whereas MOS supplies the dimensional decomposition needed for failure analysis and model iteration. The logistic-regression layer connects these two views by estimating how improvements in Prompt Following, Structural Accuracy, and Aesthetic Quality translate into expected preference shifts. This makes MEF more interpretable than evaluation pipelines that publish only rankings or only isolated scores.

Two recurrent misconceptions are directly addressed by the framework’s design. The first is that a single aggregate ranking is sufficient for T2I evaluation. MEF rejects this by providing scenario-level, prompt-level, and label-level analysis. The second is that user satisfaction can be inferred from aesthetics alone. The reported regression weights indicate that Prompt Following is generally the strongest driver of preference, although the Art scenario shows that aesthetic quality can partially compensate for prompt mismatch in that specific context.

The framework and Magic-Bench-377 are released as open-source resources. This suggests a research program centered not only on benchmarking but also on standardized, scenario-aware error analysis, with explicit room for occupational and audience-specific evaluation criteria.