GenAD-Bench Evaluation Suite

• GenAD-Bench is a comprehensive evaluation suite that uses structured six-point briefs and paired visual assets to generate end-to-end video advertising outputs.
• It integrates multimodal inputs—text prompts and image priors—with outputs such as storyboard JSON and synchronized audiovisual clips to ensure brand, product, and demographic fidelity.
• The benchmark employs balanced synthetic datasets and rigorous automated and human evaluations, setting a new standard for generative video advertising research.

GenAd-Bench is a purpose-built, end-to-end evaluation suite for generative video advertising, introduced in "Co-Director: Agentic Generative Video Storytelling" (Song et al., 27 Apr 2026). It comprises 400 richly annotated scenarios of fictional products for personalized advertising. Each scenario is defined by a compact six-point brief—brand, product, target gender, age, location, and interest—and is paired with ground-truth reference visuals consisting of a logo and a product image. Within the Co-Director framework, GenAd-Bench serves as the evaluation substrate for end-to-end systems that must transform multimodal priors into storyboarded, audiovisual advertisements rather than isolated clips (Song et al., 27 Apr 2026).

1. Task definition and benchmark scope

GenAd-Bench evaluates generative systems at the level of complete advertising outputs. Each scenario package contains a structured six-point text prompt, two reference images, and expected outputs for a valid submission: a scene-by-scene storyboard in JSON with a narrative script, and four synchronized video shots with integrated audio, for a total duration of approximately 12 seconds (Song et al., 27 Apr 2026).

The benchmark therefore spans multiple stages of the generation stack. Inputs consist of text prompt fields and visual priors, specifically the logo and the product image. Outputs include the narrative script and storyboard JSON, keyframes, diffusion-generated video clips, and a mixed audio track. A common misunderstanding is to treat GenAd-Bench as a generic text-to-video benchmark. Its task definition is narrower and more structured: it is designed for multimodal, identity-preserving, campaign-style video generation in which brand assets, product depiction, demographic targeting, and audiovisual coherence are jointly assessed.

The dataset is organized around 50 fictional brands and 4 products per brand, yielding 200 unique products. Each product is instantiated twice—once with a “stereotypical” demographic targeting and once with an “unconventional” demographic—producing 400 total scenarios. This construction makes demographic conditioning part of the benchmark itself rather than an optional downstream analysis.

2. Dataset composition and balancing scheme

The product inventory spans 24 macro-categories, including Home Goods, Food, Fashion, Tech, Automotive, Industrial, and Software (Song et al., 27 Apr 2026). The sampling is explicitly balanced in some high-frequency categories—Home Goods with 24 products, Food with 20, Fashion with 20, and Tech with 20—while still retaining lower-frequency but challenging domains such as Automotive with 8 products and Baby Care with 4. This suggests that GenAd-Bench is intended to test both breadth and category-specific difficulty, rather than merely maximize topical uniformity.

Scenario balancing is specified at the demographic level. Annotation guidelines impose exactly 200 male and 200 female scenarios, and the age brackets are equally split among Gen Z (18–24), Millennials (25–34), Gen X (35–50), and Seniors (50+). Geographic granularity is also substantial: 183 unique cities or regions across six continents, with particular emphasis on Europe, which accounts for 158 scenarios, and Asia, which accounts for 108 scenarios.

The benchmark’s paired-scenario construction is illustrated by the example product “Grault-Heel Crystal Stiletto” in the Fashion category, ID 34. Its stereotypical configuration targets a female, age 25–34, in Paris, with the interest “high-fashion influencers,” while its unconventional counterpart targets a male, age 50+, in Mumbai, with the interest “Bollywood costume designers.” The benchmark thus embeds demographic counterpoint directly into scenario design, allowing evaluation under both expected and noncanonical targeting regimes.

3. Scenario synthesis and annotation pipeline

The creation pipeline begins with an LLM, Gemini 3 Pro, which synthesized 50 brand names and wrote four diverse product descriptions per brand, explicitly requesting variety in form factors, from jewelry to subscription services (Song et al., 27 Apr 2026). For each product, the same LLM generated one “stereotypical” and one “unconventional” target persona. A text-to-image model, Nano Banana Pro, then created vector-style logos and product renderings conditioned on the textual descriptions.

Human quality control is integral to the asset-generation process. Fifteen percent of the visuals were flagged for unnatural or memorized content and regenerated until all assets were unique and legally unencumbered. This suggests that GenAd-Bench is designed not only as a technical benchmark but also as a copyright-safe scenario set for comparative research.

The benchmark’s annotation philosophy is therefore synthetic but curated. Because products, logos, and demographic settings are fictional and programmatically generated, GenAd-Bench avoids dependence on commercial brand assets while preserving the operational structure of real advertising briefs. At the same time, the human QC stage prevents the benchmark from being reducible to a purely automatic data-generation pipeline.

4. Evaluation protocol and formal metrics

At its core, GenAd-Bench uses a multimodal LLM, Gemini 3 Pro, as an automated judge, supplemented by human studies for calibration (Song et al., 27 Apr 2026). Each generated video $F$ is scored along four axes, each on a 0–100 scale, and the final report aggregates these into an overall average.

Visual Asset Fidelity (VAF) measures how faithfully the user’s logo and product reference appear in the video. Let $I_r$ be the reference image and $\hat I_t$ be the set of rendered frames. The benchmark defines

$$\mathrm{VAF} = \frac{1}{N}\sum_{i=1}^{N} \mathrm{sim}_{\mathrm{CLIP}}(I_r, \hat I_i)\times 100$$

where $\mathrm{sim}_{\mathrm{CLIP}}(\cdot,\cdot)$ is a vision-language embedding cosine similarity and $N$ is the number of sampled frames.

Demographic Alignment (DA) evaluates whether the video caters to the six-point prompt. For the four demographic dimensions $$d \in \{\text{gender}, \text{age}, \text{location}, \text{interest}\}$$, the benchmark defines

$$\mathrm{DA} = \frac{1}{4}\sum_{d} \mathbf{1}[\mathrm{match}_d]\times 100$$

where $\mathrm{match}_d$ is an MLLM judgement that the chosen demographic appears correctly, for example that actors match the age bracket and settings match the location.

Marketing Appeal (MA) operationalizes the AIDA hierarchy—Attention, Interest, Desire, Action—in a single composite:

$$\mathrm{MA} = w_1\cdot \mathrm{Hook} + w_2\cdot \mathrm{Demonstration} + w_3\cdot \mathrm{EmotionalResonance}$$

Each subscore is 0–100 as judged by the MLLM. In practice, the benchmark uses uniform weights, $I_r$0, and rescales the result to $I_r$1.

Visual Quality (VQ) penalizes diffusion artifacts, temporal flicker, and physics violations. Sampling $I_r$2 frames and letting $I_r$3 be an artifact penalty in $I_r$4, the metric is

$I_r$5

The protocol combines automated scoring with calibration by human raters. Human evaluation uses 50 validation scenarios, each judged by 5 independent raters on a 1–5 Likert scale for the same four dimensions. Inter-rater agreement is reported as Krippendorff’s $I_r$6 between 0.58–0.64 and Cohen’s $I_r$7 around 0.46–0.53. Alignment with the MLLM judge is quantified by Pearson $I_r$8 for VAF, $I_r$9 for DA and MA, Spearman $\hat I_t$0–$\hat I_t$1, and mean absolute error of 0.77–1.02 points when mapped to the 1–5 scale. A plausible implication is that the automated judge is intended as a scalable proxy rather than as a replacement for human assessment.

5. Benchmark results and comparative performance

The reported GenAd-Bench validation results compare Co-Director against commercial systems, open-source systems, and other multi-agent pipelines (Song et al., 27 Apr 2026). All scores are reported on the same 0–100 scale for VAF, DA, MA, VQ, and their average.

At the lower end of the reported range, the proprietary “Talking-head” baseline scores 23.2 VAF, 16.2 DA, 19.5 MA, 29.5 VQ, and 22.1 average. HeyGen improves to 42.9, 59.3, 39.5, 45.0, and 46.7 respectively. Kling 3.0 Omni reports 62.0 VAF, 70.3 DA, 56.0 MA, 45.3 VQ, and 58.4 average. Veo 3.1 reports 60.0, 80.8, 63.2, 50.5, and 63.6. Wan 2.6 reports 67.0, 71.5, 62.5, 58.9, and 65.0. AniMaker reports 53.1, 81.3, 60.3, 53.9, and 62.2. MovieAgent reports 61.2, 81.3, 66.4, 52.4, and 65.3.

Among agentic baselines, the Base Agentic Pipeline with $\hat I_t$2 reports 68.5 VAF, 78.4 DA, 67.1 MA, 59.9 VQ, and 68.5 average. Random Search with $\hat I_t$3 improves to 77.0, 85.8, 75.6, 64.1, and 75.7. Co-Director with $\hat I_t$4 attains the highest reported scores: 82.1 VAF, 91.4 DA, 82.0 MA, 70.2 VQ, and 81.4 average.

Beyond these quantitative results, the paper states that qualitative galleries demonstrate that Co-Director consistently preserves brand and product identity across dynamic plots, lighting shifts, and camera angles, whereas shorter, static-script pipelines struggle to maintain this identity. Within the logic of GenAd-Bench, identity preservation is not a peripheral quality criterion: it is central to both VAF and the broader notion of advertising validity.

6. Access, integration, and research role

GenAd-Bench is distributed through a GitHub repository at https://github.com/co-director-agent/GenAd-Bench and a project page at https://co-director-agent.github.io/ (Song et al., 27 Apr 2026). Each scenario folder contains prompt.json, logo.png, product.png, ground_truth_video.mp4, and validation_metadata.json. The paper provides a minimal Python loading pattern: $\hat I_t$5

The usage guidelines emphasize several best practices. Pipelines should always condition on both text and image priors to avoid copycat hallucinations. New creative-steering algorithms should use GenAd-Bench’s hillclimbing split of 200 scenarios for tuning, and evaluation should be restricted to the held-out validation split consisting of 160 in-domain and 40 out-of-domain scenarios. Any learned reward predictor should be calibrated against the MLLM-judge rubrics and, if possible, validated with a small human panel to adjust for perceptual biases, especially for VQ. Reported results should include the four per-dimension scores—VAF, DA, MA, VQ—and the overall average, together with at least one qualitative example showcasing identity preservation under novel narrative arcs.

GenAd-Bench occupies a specific position within generative video research. It is not a benchmark for unconstrained cinematic generation, nor is it limited to frame quality in isolation. Its design couples multimodal input conditioning, demographic targeting, narrative planning, and audiovisual delivery under a single protocol. By combining a carefully balanced, copyright-safe scenario set with rigorous, multimodal evaluation, GenAd-Bench establishes a new standard for end-to-end generative video advertising research. The open-source release and clear protocol ensure that future systems can be directly compared on both quantitative metrics and narrative quality.