Magic-Bench-377: Evaluation for T2I Models

• Magic-Bench-377 is a benchmark that uses 377 multi-labeled prompts to evaluate text-to-image models across real user scenarios and diverse capabilities.
• It employs a dual evaluation methodology combining ELO for overall ranking and MOS for detailed scoring on prompt following, structural accuracy, and aesthetics.
• The framework enhances validity by integrating multi-capability prompting to capture capability coupling and ensure balanced coverage of practical applications.

Magic-Bench-377 is MEF’s core, application-oriented evaluation set for text-to-image models. It contains 377 text prompts constructed under a structured taxonomy and annotated at the label level to simultaneously probe real user scenarios and objective capabilities. Each prompt is multi-labeled with 1–4 capability labels and exactly 1 application-scenario label, so that a single prompt evaluates multiple skills at once rather than isolating one capability per item. Within MEF, the benchmark is used jointly with ELO-based overall ranking and dimension-specific MOS scoring, providing both leaderboard-level comparison and fine-grained diagnosis of Prompt Following, Structural Accuracy, and Aesthetic Quality (Dong et al., 22 Sep 2025).

1. Definition and rationale

Magic-Bench-377 was introduced to address two limitations attributed to earlier text-to-image evaluation practice. First, prior benchmarks are described as either enumerating capabilities without coherent structure or focusing narrowly on compositional subskills. Second, they are described as lacking an application-scenario perspective, which limits external validity: a model that performs well on synthetic capability drills may still underperform in film, art, or design workflows. Magic-Bench-377 therefore integrates five real application scenarios with a principled capability taxonomy spanning elements, element compositions, and text expression forms (Dong et al., 22 Sep 2025).

The benchmark’s central design decision is multi-capability prompting. Each prompt is annotated with one scenario label and 1–4 capability labels, so a single item can simultaneously test, for example, number, color, and spatial relations. The benchmark is therefore intended to capture capability coupling in realistic use cases and to reduce the over-optimism that can arise when capabilities are tested in isolation. In MEF, each prompt is used to generate four images per model for both ELO and MOS evaluation, and the same prompt set underlies both the overall ranking and the label-level diagnostic analysis (Dong et al., 22 Sep 2025).

A plausible implication is that Magic-Bench-377 is designed less as a narrow stress test for one failure mode than as a structured sampling of practical prompt regimes. The benchmark is explicitly described as supporting label-level assessment while ensuring balanced coverage of both user scenarios and capabilities.

2. Taxonomy and prompt construction

Magic-Bench-377 is organized by a structured taxonomy with two principal viewpoints: application scenarios from the user perspective, and objective capabilities from the model perspective. The scenario layer contains five categories.

Scenario	Share of prompts
Film	20%
Art	21%
Entertainment	12%
Aesthetic Design	25%
Functional Design	22%

The capability layer is divided into three branches. The first branch, Element, contains Entity; Entity Description, which includes Quantity, Attribute, Relation, and Action/State; and Image/Visual Description, which includes Style, Aesthetic, and Atmosphere. The second branch, Element Composition, contains Multi-Entity Feature Matching, Layout & Typography, and Anti-Realism. The third branch, Text Expression Form, contains Negation, Pronoun Reference, and Consistency (Dong et al., 22 Sep 2025).

The reported label coverage is as follows.

Capability label	Fraction of prompts
Quantity	4%
Attribute	9%
Relation	12%
Action/State	10%
Style	64%
Aesthetic	35%
Atmosphere	6%
Multi-Entity Feature Matching	8%
Layout & Typography	5%
Anti-Realism	8%
Negation	2%
Pronoun Reference	1%
Consistency	2%

Because each prompt can have multiple capability labels, these percentages sum to more than 100%. The contributors responsible for benchmark construction are described as professional designers, experienced prompt writers or AI enthusiasts, and expert annotators. The stated goal of this curation process is clarity, fairness, and diversity within labels, including varied textures, varied relation types, and varied style families. Prompts are also described as embedding multiple test points per item, which supports later decomposition into label-level analyses (Dong et al., 22 Sep 2025).

3. Evaluation methodology within MEF

MEF uses Magic-Bench-377 as the common substrate for a joint evaluation protocol combining ELO and MOS. ELO is used for overall ranking through pairwise preference judgments, while MOS is used for absolute, dimension-specific scoring on Prompt Following, Structural Accuracy, and Aesthetic Quality. The benchmark therefore couples a relative ranking mechanism with absolute diagnostic scoring, rather than treating them as interchangeable (Dong et al., 22 Sep 2025).

For ELO, comparison is performed on MagicArena in a double-blind setting, with two anonymized images displayed side by side and four possible outcomes: left wins, right wins, both good, or both bad. The sampling strategy prioritizes pairs with fewer historical matches and favors pairs with estimated 50% win probability in order to reduce confidence intervals efficiently. Prompts are sampled uniformly from Magic-Bench-377, and one of the four generations for each model is sampled randomly. MEF estimates preferences with Bradley–Terry maximum likelihood and then linearly transforms the coefficients to ELO, with a baseline model fixed at 1,000. The framework also reports a prompt-level decomposition,

$$\mathrm{ELO}_{\text{model}} \approx \sum_{i=1}^{377} \mathrm{ELO}_{\text{prompt } i}, \quad \text{err} < 5\%.$$

Leaderboard stability is associated with a confidence-interval width of at most 20 ELO points and at most 3 ELO change per match, typically requiring more than 4,000 matches per model (Dong et al., 22 Sep 2025).

MOS uses a 1–5 scale for the three base dimensions. Four independent samples are generated per prompt per model, and two experts score all models’ outputs side by side for each prompt. The stated reason for repeated generation is that self-variation drops from approximately 30% to less than 5%, with $p < 0.01$. Confidence estimation for repeated-measures MOS is reported as

$$\mathrm{M} = \frac{1}{n} \sum_{i=1}^{n} \left( \frac{1}{k} \sum_{j=1}^{k} a_{i,j} \right), \quad \mathrm{var}(\mathrm{M}) = \frac{1}{n^2 \times k} \sum_{i=1}^{n} \mathrm{var}(a_i).$$

MEF further links MOS dimensions to user preference via multivariate logistic regression on standardized MOS features, reporting average changes in absolute win rate from a one-standard-deviation increase in each dimension rather than raw coefficients (Dong et al., 22 Sep 2025).

4. Annotation protocol, reliability, and quality control

The evaluation protocol distinguishes expert mode from public mode. Expert mode uses a trained panel typically numbering fewer than 20 experts, while public mode uses more than 1,000 participants from design, media, film and television, and AI enthusiast communities. Expert qualification requires at least 85% agreement with the cohort and Kappa greater than 0.8, and the system includes periodic recertification and re-auditing (Dong et al., 22 Sep 2025).

Quality control combines procedural and statistical checks. Five percent anchor items are inserted to detect inconsistent raters, and temporal behavior analysis is used to identify anti-cheating signals such as extreme speed or repetitive response patterns. In expert mode, 25% of assessments are re-audited, and noncompliant experts may be temporarily suspended. The reporting is described as including raw data, 95% confidence intervals, and significance testing at $p < 0.01$ (Dong et al., 22 Sep 2025).

The MOS protocol also includes explicit decoupling instructions. Raters are asked to focus on the dimension being scored unless a severe issue in another dimension prevents interpretation. The reported inter-dimensional Pearson correlations are all below 0.3, with Structural Accuracy versus Aesthetic Quality around 0.27, and this is presented as evidence that the three MOS dimensions are near-decoupled in practice. This suggests that Magic-Bench-377 is intended არა only as a ranking benchmark but also as a measurement instrument for disentangling types of generation error (Dong et al., 22 Sep 2025).

5. Reported results and diagnostic findings

On the expert-mode leaderboard, based on 62,736 valid matches, GPT-4o is reported at approximately 1205 ELO overall and Seedream 3.0 at approximately 1178, followed by Imagen 3 at approximately 1100, Ideogram 3.0 at approximately 1078, Luma at approximately 1076, FLUX.1 Kontext [pro] at approximately 1062, Reve Image 1.0 at approximately 1061, Ideogram 2.0 at approximately 1054, Seedream 2.1 at approximately 1032, Recraft V3 Raw at approximately 1021, Flux 1.1 [pro] at approximately 1011, and Midjourney V6.1 at approximately 1000. In public mode, based on 106,287 valid matches, Seedream 3.0 ranks first overall at approximately 1084, with GPT-4o close behind at approximately 1067. The ordering is described as broadly consistent with expert mode but compressed, with only about 78 ELO points from top to bottom versus about 200 in expert mode (Dong et al., 22 Sep 2025).

Dimension-wise MOS scores for six representative models show GPT-4o leading Prompt Following with 4.52 and Structural Accuracy with 4.23, while Seedream 3.0 scores 4.23 and 3.87 on those dimensions respectively and leads Aesthetic Quality with 3.39. Midjourney V6.1 is also reported at 3.38 on Aesthetic Quality, despite weaker Prompt Following at 3.36 and Structural Accuracy at 3.31. Imagen 3 records 3.96 on Prompt Following and 3.79 on Structural Accuracy; Ideogram 3.0 records 3.91 and 3.71; Flux 1.1 [pro] records 3.48 and 3.41 (Dong et al., 22 Sep 2025).

The benchmark also supports model-specific diagnosis. GPT-4o is described as strongest in semantics and structure, with strengths in multi-entity matching, layout and typography, anti-realism, negation, pronoun reference, and consistency. Seedream 3.0 is described as balanced across all dimensions, with top aesthetics and strengths on textures, actions, and interactions. Midjourney V6.1 is described as an aesthetics specialist with weaker Prompt Following and structural performance. Ideogram 3.0 is described as strong on text rendering and layout tasks. Persistent hard cases across models are reported to include complex relational reasoning, quantity, element composition, negation, co-reference, and consistency (Dong et al., 22 Sep 2025).

The logistic-regression analysis connects these observations to user preference. From a 50% baseline win rate, a one-standard-deviation increase in Prompt Following corresponds to a reported gain of $+12.5\%$ for general users, $+37.7\%$ for experts, and $+26.0\%$ for designers. The corresponding gains for Structural Accuracy are $+0.7\%$, $+14.2\%$, and $+10.0\%$, while Aesthetic Quality yields $p < 0.01$0, $p < 0.01$1, and $p < 0.01$2. By scenario, Prompt Following is reported to receive the lowest weight in Art among the five scenarios, which explains why a model with stronger aesthetics can rank relatively higher in Art despite weaker Prompt Following. MEF ELO is also reported to correlate strongly with the third-party Artificial Analysis leaderboard, with Pearson correlation around 0.8, despite using only 377 prompts (Dong et al., 22 Sep 2025).

6. Release, scope, and distinction from similarly named benchmarks

Magic-Bench-377 is released as part of MEF and is described as fully open-source, with the dataset hosted at https://huggingface.co/datasets/ByteDance-Seed/MagicBench. The recommended protocol is explicit: generate four images per prompt for each model, conduct double-blind head-to-head ELO evaluation with Bradley–Terry maximum likelihood and bootstrap confidence intervals, score MOS with two trained experts on the three core dimensions, and fit multivariate logistic regression to interpret how MOS dimensions affect ELO win probability. Quality assurance is intended to follow the same anchor-item, temporal-analysis, and expert-qualification procedures used in the paper’s evaluation (Dong et al., 22 Sep 2025).

The benchmark’s stated limitations are also specific. A set of 377 prompts cannot exhaust combinatorial cases, so specialized supplementary sets may still be needed for rare scenarios. Human-in-the-loop evaluation remains costly, motivating future work on automated methods. The current scope is single-image text-to-image generation rather than broader multimodal conditioning or output formats. The paper also notes that manual curation and evaluator composition may introduce residual biases, although scenario and label distributions are reported to increase transparency (Dong et al., 22 Sep 2025).

A common source of confusion is the similarity between Magic-Bench-377 and MageBench. They are distinct benchmarks. MageBench is a multimodal agent benchmark oriented around Vision-in-the-Chain reasoning, not a text-to-image evaluation set, and the MageBench paper states explicitly that “Magic-Bench-377” does not appear anywhere in that work. The correct MageBench count is 483 total scenarios across WebUI, Sokoban, and Football, rather than 377 (Zhang et al., 2024). This distinction is important because the two benchmarks differ in modality, task structure, metrics, and intended research use: Magic-Bench-377 is scenario-grounded evaluation for T2I models, whereas MageBench evaluates multimodal agents in interactive environments.