E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought

Abstract: Generative AI is widely used to create commercial posters. However, rapid advances in generation have outpaced automated quality assessment. Existing models emphasize generic esthetics or low level distortions and lack the functional criteria required for e-commerce design. It is especially challenging for Chinese content, where complex characters often produce subtle but critical textual artifacts that are overlooked by existing methods. To address this, we introduce E-comIQ-ZH, a framework for evaluating Chinese e-commerce posters. We build the first dataset E-comIQ-18k to feature multi dimensional scores and expert calibrated Chain of Thought (CoT) rationales. Using this dataset, we train E-comIQ-M, a specialized evaluation model that aligns with human expert judgment. Our framework enables E-comIQ-Bench, the first automated and scalable benchmark for the generation of Chinese e-commerce posters. Extensive experiments show our E-comIQ-M aligns more closely with expert standards and enables scalable automated assessment of e-commerce posters. All datasets, models, and evaluation tools will be released to support future research in this area.Code will be available at https://github.com/4mm7/E-comIQ-ZH.

• The paper introduces E-comIQ-18k, a human-aligned dataset of 18,000 Chinese e-commerce posters annotated across object, text, layout, and background quality, achieving 0.858 overall Krippendorff’s alpha during calibration.
• The paper develops E-comIQ-M through supervised fine-tuning and GRPO, raising overall SRCC from 0.119 to 0.433 and showing that domain-specific training outperforms larger general-purpose multimodal evaluators.
• The paper demonstrates that text is the dominant quality bottleneck, accounting for 44.8% of weakest-link cases, while OCR metrics often miss stroke-level corruption that human reviewers detect.
• What annotation and evaluation methods best measure Chinese typography and stroke-level text errors in generated posters?
• How does GRPO improve fine-grained visual quality assessment beyond supervised fine-tuning alone?
• Can E-comIQ-M generalize to other languages, poster aspect ratios, and commercial design formats?
• How should reference-based metrics such as CLIP, DINO, and LPIPS be combined with human-aligned evaluators?
• Find recent papers about human-aligned multimodal evaluation of AI-generated images.

Motivation and problem statement

Generative models now routinely produce e-commerce posters, but quality assessment has not kept pace with generation capability. Existing image quality assessment (IQA) tools target low-level distortions (blur, noise, compression) or generic aesthetics, and multimodal LLM evaluators are trained on open-domain data. Neither captures the functional criteria that determine whether a poster is commercially usable — product visibility, copy correctness, typographic legibility, layout hierarchy. The gap is most severe for Chinese content: dense typography combined with complex character sets produces subtle stroke-level rendering errors that general-purpose evaluators systematically miss, while OCR-based metrics can score visually corrupted text as correct.

The paper addresses this with three artifacts: E-comIQ-18k, an 18,000-image dataset of Chinese e-commerce posters with multi-dimensional expert scores and expert-calibrated Chain-of-Thought (CoT) rationales; E-comIQ-M, a Qwen2.5-VL-7B-based evaluator trained via SFT plus GRPO; and E-comIQ-Bench, a 500-case benchmark for comparing text-to-image systems on poster generation.

Dataset construction

E-comIQ-18k draws from six sources: merchant HQ/LQ photos (27.8% each), open-source posters from Autoposter (16.7%), AI-generated posters produced by GPT-4o conditioned on product cutouts (11.1%), AI-edited compositions built with Flux plus template-rendered Chinese copy (11.1%), and professionally designed posters (5.6%). Splits are 15k/2k/1k train/validation/test with balanced source and quality distributions.

Quality is decomposed into four dimensions annotated on a [1, 5] continuous scale with three anchored tiers (poor, good, excellent), plus multi-label issue tags from a per-dimension checklist. A notable design choice separates Object from Text: text printed on product packaging counts toward Object, while overlaid marketing copy defines the Text dimension.

Annotation reliability is substantial. Six experts first cross-annotate a 1,000-image calibration set to a stable overall Krippendorff's $\alpha$ of 0.858 (per-dimension: 0.721–0.877); the remaining images are partitioned without overlap, with 10% random re-sampling for drift control. CoT rationales averaging over 800 Chinese characters are generated by Qwen2.5-VL-Max from expert scores and tags, then span-level edited by the original annotator via a NER-style interface. The reported average character-level edit rate is 32.3% (maximum 83.4%), which is worth emphasizing: it indicates that raw MLLM rationales require heavy human correction, and that dataset fidelity rests on this supervised editing step rather than on the generator alone.

Two statistical findings justify the multi-dimensional design. Inter-dimension Pearson correlations are weak (mean $\rho \approx 0.24$), so holistic scores cannot substitute for sub-scores. And in a "weakest link" analysis over images with any dimension below 3.0, Text is the bottleneck in 44.8% of cases and correlates most strongly with overall quality ($\rho = 0.67$) — evidence that text rendering, not aesthetics, is the central failure mode for Chinese posters.

The E-comIQ-M evaluator

E-comIQ-M fine-tunes Qwen2.5-VL-7B-Instruct end-to-end. Stage one is full-parameter SFT on all 15k training samples against expert scores and CoT rationales, producing structured JSON outputs (four sub-scores plus overall). Stage two applies GRPO on a curated hard subset — the worst 3k samples by SFT MSE, stratified across sources — using a composite reward combining:

• an accuracy reward $R_{\text{acc}}$: fraction of dimensions within tolerance $\tau = 0.2$, down-weighted by a 0.7 factor when a prediction crosses an expert-defined quality tier;
• a distribution reward $R_{\text{dist}}$: exponential penalty on the Euclidean distance between predicted and ground-truth 4D sub-score vectors;
• a binary format reward for parseable JSON.

The reward weights ($\lambda_{\text{score}} = 0.65$) and hyperparameters are tuned via ablation; performance is stable across hard-subset sizes from 1k to 5k.

Evaluation results

On the held-out test set, baseline performance confirms the domain gap. Traditional NR-IQA models (MUSIQ, SPAQ) achieve near-zero or negative PLCC/SRCC. Frontier MLLMs — GPT-4o, Gemini 2.5 Pro, Claude Sonnet 4.5, Grok-4 — reach only ~0.18–0.26 overall SRCC, consistently weaker on Text than Background. Specialized evaluators such as Q-Align and Q-Insight fare no better on this data.

Domain-specific supervision dominates model scale. SFT alone lifts Qwen2.5-VL-7B's overall SRCC from 0.119 to 0.346 and [email protected] from 29.3% to 51.0%, already surpassing GPT-4o and Gemini. Adding GRPO yields E-comIQ-M at overall SRCC 0.433 / PLCC 0.425 and [email protected] of 55.6%, with the largest gains on Text (SRCC 0.392 vs. 0.283 after SFT alone). Ablations show GRPO without SFT collapses (overall SRCC 0.158), and the distribution reward term adds roughly 0.05–0.07 SRCC over an accuracy-only reward — supporting the claim that aligning sub-score geometry matters beyond pointwise accuracy.

Benchmarking generative models

E-comIQ-Bench comprises 500 cutout–prompt pairs spanning seven product categories, evaluated with Seedream 4.0, Qwen-Image-Edit, GPT-4o, Gemini 2.5 Flash Image, and Flux-Kontext-max, with original merchant posters as references. Human designer panels rate every output alongside E-comIQ-M predictions, complemented by reference-based diagnostics: DINO similarity, LPIPS, and CLIP score for subject fidelity; phrase-level F1 and normalized Levenshtein similarity for text accuracy.

Three results stand out:

1. The strongest generator slightly exceeds average human merchant-poster quality on the Overall dimension, but no system clearly surpasses human design; Text remains the bottleneck across systems.
2. E-comIQ-M reproduces system rankings and per-dimension strength profiles despite moderate absolute correlation (~0.34 PLCC/SRCC) in this out-of-domain setting — adequate for comparative benchmarking, though not for precise absolute scoring.
3. OCR-style text metrics disagree sharply with human judgments. Gemini and GPT-4o attain high phrase F1 (0.56–0.86) and character similarity (0.62–0.91) despite low human text scores (1.45–2.83), because stroke-level corruption still maps to intended phrases under OCR. Flux shows the opposite extreme: phrase F1 of 0.10 with a low human text rating of 1.48, consistent with near-total text-generation failure. This directly validates the paper's premise that reference-free, human-aligned evaluation is necessary for Chinese commercial imagery.

Limitations and open questions

The authors concede two material constraints. First, as a non-reference evaluator, E-comIQ-M cannot measure subject identity fidelity directly; the benchmark must rely on auxiliary DINO/LPIPS/CLIP diagnostics against the original poster, which themselves depend on segmentation quality. Second, alignment with human judgment remains moderate — around 0.34 correlation on out-of-domain benchmark data versus 0.43 in-distribution — leaving headroom before automated scores can substitute for expert review in high-stakes decisions. Additional unstated assumptions include reliance on a single annotator per image outside the calibration set (drift controlled only by 10% sampling), and evaluation restricted to square 800×800 generations under default inference settings, so conclusions may not transfer to other aspect ratios or tuned configurations.

Conclusion

This work supplies the missing measurement infrastructure for Chinese e-commerce poster generation: a multi-dimensional annotation standard with strong inter-annotator agreement, an 18k-sample dataset with expert-verified CoT rationales, a two-stage SFT+GRPO evaluator that substantially outperforms both frontier MLLMs and existing IQA specialists on domain-specific criteria, and a reproducible benchmark showing that text rendering — invisible to OCR metrics — remains the dominant limitation of current generative systems. The moderate human-model correlation on out-of-domain data identifies score calibration as the concrete open problem this line of work leaves unresolved.