---
title: 'PosterBench: A Benchmark for Poster Generation'
url: https://www.emergentmind.com/topics/posterbench
type: topic
---

# PosterBench: A Benchmark for Poster Generation

PosterBench is a benchmark for evaluating text-to-image systems that generate complete graphic-design posters from textual creative briefs. Introduced with PosterReward, it evaluates integrated poster quality through a specialized reward model that jointly considers typography, layout, prompt compliance, image quality, generative artifacts, and aesthetics. It is distinct from PosterRewardBench, which evaluates the accuracy of reward models and multimodal language models as poster-quality assessors. PosterBench is therefore a generation benchmark whose principal scores are produced automatically by PosterReward rather than directly by human raters [2603.29855].

## 1. Scope and research context

PosterBench addresses limitations of general text-to-image evaluation, which commonly emphasizes global image aesthetics, image–text similarity, or broad semantic alignment. Such criteria are insufficient for poster generation because posters must satisfy simultaneous textual, spatial, visual, and communicative constraints.

The benchmark treats poster creation as a multidimensional design problem involving:

- exact rendering of required words;
- spelling, omissions, and extraneous text;
- font style, weight, color, capitalization, and scale;
- kerning, leading, alignment, and text placement;
- integration of typography into the composition;
- layout and compositional structure;
- visual polish;
- absence of AI-generated artifacts;
- compliance with the requested creative brief.

PosterBench is designed to evaluate complete poster artifacts rather than isolated subtasks such as OCR, bounding-box prediction, or aesthetic preference. Its scoring model is intended to distinguish a visually attractive image with incorrect text from a poster that satisfies both design and content requirements.

The benchmark belongs to a broader progression in poster-generation research. Early academic poster systems modeled content volume, panel geometry, and graphical-element placement with probabilistic graphical models using paired papers and posters [1702.06228]. PosterLayout later addressed content-aware visual–textual layout over non-empty canvases, using categorized bounding boxes and saliency-aware metrics [2303.15937]. PosterLlama reformulated layout generation through HTML serialization and multimodal language modeling [2404.00995]. P2Peval introduced source-grounded and importance-weighted evaluation for paper-to-poster generation [2505.17104]. PosterIQ expanded poster assessment toward typography, composition, metaphor, intention, and human-centred design reasoning [2603.24078]. PosterBench differs from these settings by evaluating generated poster images through a poster-specific reward model and by explicitly measuring repeated-generation quality and stability.

## 2. Benchmark composition and generation protocol

PosterBench contains 250 distinct poster-generation prompts:

- 100 cinematic prompts;
- 150 non-cinematic prompts covering diverse commercial and public-service graphic-design themes;
- eight generated images per prompt and per evaluated model.

A complete model evaluation therefore nominally produces 2,000 generated images. The prompts were randomly sampled from PosterRewardBench and stratified into the cinematic and non-cinematic categories. The benchmark is presented as a fixed evaluation set; no train, validation, or test split is specified.

Cinematic prompts are evaluated at a common vertical resolution of $832 \times 1216$, approximately corresponding to a $2{:}3$ movie-poster aspect ratio. Non-cinematic prompts use target resolutions $(H_i,W_i)$ derived from associated reference ground truths. These targets may be portrait, landscape, or square, so models are evaluated under heterogeneous output geometries rather than a single preferred aspect ratio.

Resolution handling depends partly on model accessibility:

- open-weight models such as Qwen-Image and Flux are conditioned to generate directly at the target resolution;
- proprietary systems with discrete resolution choices use approximate aspect-ratio mappings;
- Nano-Banana maps targets to its nearest supported aspect ratio, including $1{:}1$, $2{:}3$, and $16{:}9$;
- GPT-Image-1 uses portrait, landscape, and square resolution buckets;
- Seedream-4.5 is generated at target 2K resolution and subsequently resized to the standard benchmark resolution.

These procedures improve comparability across systems but do not eliminate implementation differences. Diffusion steps, guidance scales, samplers, random seeds, and other model-specific inference parameters are not fully standardized in the supplied benchmark description. Results should therefore be interpreted as comparisons under the authors’ reported implementation settings.

The evaluated systems comprise 13 models.

| Category | Models |
|---|---|
| Closed-source | Nano-Banana-Pro, Seedream-4.5, Nano-Banana, Seedream-4.0, GPT-Image-1, Seedream-3.0 |
| Open-source | Qwen-Image-2512, Qwen-Image, Z-Image-Turbo, Flux.2-klein-9B, Flux.1-krea-dev, Flux.1-dev, SD3.5-L |

For every prompt–model pair, the protocol is:

1. generate eight poster candidates;
2. use the assigned resolution or closest permitted geometry;
3. score every candidate with PosterReward;
4. aggregate the eight scores for each prompt;
5. aggregate results over the 250 prompts.

The eight-sample design permits evaluation of both typical output quality and best-case performance. It also exposes stochastic instability that would be hidden by evaluating only one generation.

## 3. PosterReward evaluation model

PosterBench uses PosterReward as its principal evaluator. PosterReward is a cascaded discriminative reward model trained specifically for poster assessment. Its first stage analyzes the prompt–image pair, and its second stage converts the image and analysis into a scalar score.

The analysis module is initialized from Qwen3-VL-8B. The scoring module receives the generated image, prompt, and analysis text, and replaces the final Qwen3-VL-8B layer with a two-layer SiLU-connected multilayer perceptron. PosterReward-Lite omits the analysis module for faster inference, whereas the principal PosterBench results use the full PosterReward model.

The analysis covers five dimensions:

1. **Fundamental Image Integrity**: clarity, sharpness, exposure, blur, pixelation, noise, and related technical defects.
2. **AI Artifact and Realism Evaluation**: malformed objects, anatomical failures, incoherent blending, unrealistic textures, and other generative errors.
3. **Typographical Precision Analysis**: required text, spelling, omissions, additions, font family and style, color, weight, capitalization, kerning, leading, alignment, scale, legibility, and integration with the composition.
4. **Visual Prompt Interpretation**: compliance with requested subjects, objects, setting, composition, layout, artistic style, mood, and color palette.
5. **Standalone Artistic Evaluation**: composition, lighting, color theory, emotional impact, creative decisions, technical execution, and overall aesthetic appeal independently of strict prompt compliance.

The benchmark’s preference hierarchy gives priority to fundamental image integrity, comprehensive prompt adherence, textual accuracy, artifact reduction and realism, and standalone aesthetics. These dimensions are analyzed separately internally, but PosterBench reports one overall PosterReward score rather than a public dimension-by-dimension score.

PosterReward was developed using poster-specific preference data because broad reward models such as ImageReward, PickScore, HPSv2, HPSv3, and UnifiedReward primarily target general image-generation preferences. OCR-only evaluation has the opposite limitation: it can reward character recognition while failing to assess typography, composition, aesthetics, or integration of text into the design.

PosterRewardBench serves as the evaluator-validation counterpart to PosterBench. It contains 1,740 human-annotated pairwise poster preferences, reviewed by four professional annotators, with pairs retained when at least three annotators agreed. PosterReward achieves 86.7% accuracy on the basic subset and 86.0% on the advanced subset in the reported table. The supplied paper also states 86.5% for the advanced subset in prose, producing an unresolved numerical inconsistency [2603.29855].

## 4. Metrics

PosterBench reports four aggregate statistics. Mean, Median, and Best-of-8 Average are higher-is-better; Std-Avg is lower-is-better.

### Mean

Mean is the average PosterReward score over the generated samples. For prompt $p$ with eight scores $s_{p,1},\ldots,s_{p,8}$, the natural prompt-level mean is:

$$
\bar{s}_p=\frac{1}{8}\sum_{j=1}^{8}s_{p,j}.
$$

The benchmark does not fully specify whether its final mean is calculated by pooling all generated images or by first averaging within prompts and then averaging over prompts.

### Median

Median is the median PosterReward score over the evaluated generations. It provides a central-tendency measure less sensitive to exceptionally high- or low-scoring samples than the mean.

### Std-Avg

Std-Avg measures stochastic generation stability. For each prompt, the standard deviation of its eight scores is calculated and then averaged:

$$
\mathrm{Std\text{-}Avg}
=
\frac{1}{N}
\sum_{p=1}^{N}
\operatorname{std}
\left(s_{p,1},\ldots,s_{p,8}\right),
$$

where $N=250$. Lower values indicate more consistent quality across repeated generations. The paper does not specify whether the population or sample convention is used for the standard deviation.

### Best-of-8 Average

Best-of-8 Average, or Bo8-Avg, measures the expected best score obtainable by selecting the highest-scoring sample among eight generations for each prompt:

$$
\mathrm{Bo8\text{-}Avg}
=
\frac{1}{N}
\sum_{p=1}^{N}
\max_{1\leq j\leq 8}s_{p,j}.
$$

Bo8-Avg is relevant to test-time generation and reranking, but it should not be interpreted as typical output quality. Mean and Median describe ordinary generation performance, while Bo8-Avg measures best-case performance under an eight-sample budget.

The benchmark does not provide direct OCR accuracy, typography-only scores, layout-only scores, human PosterBench scores, confidence intervals, statistical significance tests, or a publicly documented normalization formula for the displayed PosterReward values. PosterBench should therefore be interpreted as an automated relative-comparison protocol rather than a complete decomposition of poster quality.

## 5. Reported results

Nano-Banana-Pro achieves the highest overall results among the evaluated systems. Qwen-Image-2512 is the strongest open-source model by Mean and Median and has the lowest reported Std-Avg.

| Model | Mean | Median | Std-Avg | Bo8-Avg |
|---|---:|---:|---:|---:|
| Nano-Banana-Pro | **13.36** | **13.47** | 1.91 | **15.77** |
| Seedream-4.5 | 12.03 | 12.09 | 2.08 | 14.57 |
| Nano-Banana | 11.60 | 11.69 | 2.17 | 14.49 |
| Seedream-4.0 | 11.46 | 11.44 | 2.06 | 13.93 |
| GPT-Image-1 | 11.16 | 11.38 | **1.75** | 13.43 |
| Qwen-Image-2512 | 11.86 | 11.63 | **1.46** | 13.85 |
| Qwen-Image | 7.69 | 7.72 | 2.55 | 11.06 |
| Z-Image-Turbo | 7.65 | 7.31 | 2.18 | 10.47 |
| Flux.2-klein-9B | 7.38 | 7.66 | 3.20 | **11.67** |
| Flux.1-krea-dev | 5.00 | 5.14 | 3.59 | 9.58 |
| Seedream-3.0 | 5.01 | 5.13 | 3.66 | 9.75 |
| Flux.1-dev | 2.55 | 2.42 | 3.85 | 7.81 |
| SD3.5-L | -2.90 | -3.92 | 2.68 | 1.24 |

The reported conclusions are:

- **Nano-Banana-Pro** ranks first on Mean, Median, and Bo8-Avg.
- **Qwen-Image-2512** is the leading open-source model by Mean and Median.
- Qwen-Image-2512 has the lowest reported Std-Avg, indicating stable quality across eight generations.
- GPT-Image-1 has the lowest Std-Avg among the closed-source models, although its average quality is below Nano-Banana-Pro and the Seedream-4.x systems.
- Seedream-3.0 performs substantially worse than Seedream-4.0 and Seedream-4.5.
- Flux.2-klein-9B has a relatively large gap between Mean and Bo8-Avg, indicating occasional high-scoring generations despite weaker typical quality.
- Older or less capable systems, particularly Flux.1-dev and SD3.5-L, perform poorly on the combined demands of poster layout, text rendering, and aesthetics.

The benchmark is not a universal ranking of image-generation systems. Scores depend on prompt composition, model accessibility, resolution mapping, inference configuration, PosterReward’s learned preferences, and the source distribution inherited from PosterRewardBench.

## 6. Data provenance, limitations, and interpretation

PosterBench is derived from PosterRewardBench rather than being an entirely independently authored prompt collection. PosterRewardBench itself is constructed from poster-preference data containing approximately 0.8 million cinematic images generated from 81,000 English and 58,000 Chinese prompts, together with non-cinematic images generated from English and Chinese prompts using Qwen-Image-Lightning. Preference filtering uses HPSv3 ranking, repeated ranking rounds, agreement analysis, pairwise filtering, and validation by Gemini-2.5-Pro, GPT-5, and GLM-4.5V.

This provenance gives PosterBench a poster-specific evaluation basis, but it also introduces possible dependencies between the benchmark prompts, the preference dataset, the evaluator, and the models used during data construction. The benchmark does not provide a complete prompt list, a finer-grained taxonomy of non-cinematic posters, a training or validation split, per-category results, or detailed source metadata.

Important limitations include:

- **Automated-evaluator dependence**: PosterBench’s main scores are produced by PosterReward rather than human annotators.
- **Incomplete evaluator decomposition**: PosterReward analyzes typography, layout, prompt interpretation, artifacts, and aesthetics, but PosterBench publishes only an overall score.
- **Unspecified score calibration**: the scale and normalization of the displayed scores are not fully defined.
- **Inference heterogeneity**: model-specific samplers, diffusion steps, guidance scales, and API constraints are incompletely documented.
- **Resolution mismatch**: proprietary systems may use approximate aspect-ratio mappings rather than the exact target geometry.
- **Source dependence**: prompts are sampled from PosterRewardBench, and the evaluator is trained from related poster-preference data.
- **No direct scientific or semantic fidelity metric**: PosterBench is designed primarily for artistic and commercial poster generation, not paper-to-poster factual retention.
- **No human PosterBench score**: human evaluation is reported for PosterRewardBench and related validation studies, but not as the principal PosterBench generation metric.
- **Potential language and cultural bias**: the underlying preference ecosystem contains substantial Chinese and English material, while the exact language distribution of PosterBench is not specified.
- **Best-of-8 interpretation**: Bo8-Avg measures selection-assisted best-case quality and is not equivalent to ordinary generation quality.

PosterBench’s principal methodological contribution is the combination of poster-specific evaluation, heterogeneous target resolutions, repeated generation, and stability-sensitive reporting. Its four metrics distinguish typical quality, central tendency, stochastic variation, and best-case performance. The benchmark is therefore more informative than a single-score evaluation when the research question concerns both design quality and reliability across repeated generations.

For rigorous use, results should report Mean, Median, Std-Avg, and Bo8-Avg together; preserve the exact prompt and resolution mappings; disclose model-specific inference settings; and supplement PosterReward with independent OCR, typography, layout, and human assessments. In particular, PosterBench should not be treated as a pure OCR benchmark, a purely human-aesthetic leaderboard, or a definitive measure of general text-to-image quality. It is best understood as a fixed, automated benchmark for relative comparison of integrated poster-generation systems under an eight-sample generation protocol [2603.29855].

Source: https://www.emergentmind.com/topics/posterbench