GEM-Bench: AIR Generation Benchmark

• GEM-Bench is a benchmark for ad-injected response generation in Generative Engine Marketing, integrating native ads seamlessly into LLM responses.
• It employs curated datasets and a multi-agent framework to generate ad-injected responses while balancing informativeness, naturalness, trust, and engagement.
• The benchmark utilizes a metric ontology that quantitatively and qualitatively assesses user satisfaction and engagement through measures like accuracy, naturalness, and click-through rates.

GEM-Bench is a benchmark for ad-injected response generation within Generative Engine Marketing (GEM), an emerging ecosystem for monetizing generative engines by seamlessly integrating relevant advertisements into their responses. It is framed around the generation and evaluation of ad-injected responses (AIRs), and combines curated datasets, a metric ontology spanning user satisfaction and user engagement, and baseline systems implemented in an extensible multi-agent framework (Hu et al., 17 Sep 2025).

1. Scope and problem formulation

Generative Engine Marketing treats the output of an LLM-based assistant or AI-overview system as the monetizable surface. In this setting, a generative engine produces synthesized text rather than ranked links, and advertising is integrated natively into the generated response rather than attached as an external banner or sponsored result. GEM-Bench focuses on the AIR generation stage: given a user query, optional conversational context, an advertisement database, and an inference model, the system must return a response that integrates relevant ads while balancing informativeness, naturalness, trust, and engagement (Hu et al., 17 Sep 2025).

The benchmark formalizes the setting with an advertisement database $D$, where each $d_i \in D$ contains a bidding string, its embedding, bidding price, and product metadata such as advertiser, description, URL, and creatives. An inference model $M$ receives a query and optional context and must generate an AIR that integrates $k$ relevant ads from $D$; in the reported experiments, $k=1$ (Hu et al., 17 Sep 2025).

The surrounding GEM workflow mirrors search-engine marketing but is adapted to unstructured generated text. The stages described for the broader GEM setting are offline bidding, query processing, slot identification, performance prediction, and auction and allocation. Within that broader pipeline, GEM-Bench isolates the technical problem of generating and evaluating ad-injected responses, because standard LLM benchmarks and standard advertising benchmarks do not directly measure the trade-off between helpfulness and monetization in native LLM outputs (Hu et al., 17 Sep 2025).

A central premise is that ad insertion is a multi-objective optimization problem. AIR generation must maximize user satisfaction and user engagement simultaneously, under the constraint that the ad remains native to the response rather than appearing as a separate display unit. This suggests that naive prompt-level promotion and post hoc insertion may occupy different positions on the trade-off surface between click-oriented behavior and conversational quality.

2. Benchmark composition and dataset design

GEM-Bench contains three curated datasets spanning chatbot and search-engine scenarios. Two are chatbot-oriented, and one is search-oriented (Hu et al., 17 Sep 2025).

Dataset	Scenario	Ads / Queries
MT-Human	Chatbot	6,556 ads; 10 queries
LM-Market	Chatbot	6,556 ads; 100 queries
CA-Prod	Search Engine	2,215 ads; 120 queries

MT-Human is derived from MT-Bench. The construction process manually reviewed all 80 MT-Bench queries for ad suitability and retained the 10 first-turn queries from the humanities category. These are queries such as business etiquette and lifestyle questions that can plausibly support relevant products or services. Its ad database is borrowed from Ad-Chat and Google Topics: 25 main topics, 576 subtopics, and 10 ads per subtopic generated by GPT-3.5-Turbo, yielding 6,556 products, brands, or organizations with brand name, description, and URL fields (Hu et al., 17 Sep 2025).

LM-Market is built from LMSYS-Chat-1M. The pipeline filters to single-turn English conversations, keeps the subset labeled “Marketing,” embeds queries with text-embedding-3-small, applies BIRCH clustering to 3,758 leaf clusters, then hierarchical linkage clustering to a 20-cluster level selected for human review. GPT-4o summarizes cluster topics, and three clusters with strong advertising potential are retained: Travel Planning, Recipe Recommendation, and Software Tools Comparison. From the resulting 1,701 queries, an LLM-based ad-suitability score in $[0,10]$ is assigned, queries with score above 5 are sampled, and 100 are manually confirmed (Hu et al., 17 Sep 2025).

CA-Prod is adapted from AdsCVLR and targets search-engine AIR generation. It begins with query-ad pairs carrying binary relevance labels, removes incomplete records, retains queries with both positive and negative ads, concatenates ad metadata into text, embeds ads semantically, and applies K-means clustering. The chosen clustering produces six product-topic groups: lawn and garden equipment, slip-on shoes, modern household items, nutrition supplements, Android tablets and smartphones, and women’s dresses. Queries are assigned topics from the majority topic of their positive ads, filtered by ad-count and positive-ratio constraints, and then 20 queries are sampled from each topic, producing 120 queries associated with 2,215 unique products (Hu et al., 17 Sep 2025).

The dataset construction procedures are not merely corpus assembly. They are intended to ensure that the benchmark contains queries that are genuinely suitable for ad injection, rather than merely convenient for testing retrieval or generation. This design choice matters because many standard evaluation domains—such as math, coding, or classification—are intrinsically poor AIR settings. GEM-Bench therefore restricts itself to domains where native promotional content is at least plausibly compatible with user intent.

3. Metric ontology and evaluation methodology

GEM-Bench defines a metric ontology organized around two aspects, user satisfaction and user engagement, each decomposed into concrete metrics and paired qualitative dimensions (Hu et al., 17 Sep 2025).

Aspect	Metric	Dimensions
User Satisfaction	Accuracy	Relevance, Accuracy
User Satisfaction	Naturalness	Interruptiveness, Authenticity
User Satisfaction	Personality	Helpfulness, Salesmanship
User Satisfaction	Trust	Credibility, Bias
User Engagement	Notice	Notice, Attitude
User Engagement	Click	Notice, Click

The benchmark uses both quantitative and qualitative metrics. For a response $r=\{s_1,\dots,s_\ell\}$ with $\ell>1$ sentences, response flow is defined as

$$f(r)=\sum_{i=1}^{\ell-1}\frac{\text{sim}(s_i,s_{i+1})}{\ell-1},$$

where $d_i \in D$0 is cosine similarity between sentence embeddings. This measures semantic continuity between adjacent sentences. Response coherence is defined as

$d_i \in D$1

where $d_i \in D$2 is the mean embedding of all sentences and serves as a proxy for the response’s main idea. These are global metrics over the entire AIR (Hu et al., 17 Sep 2025).

Because a single ad sentence may have only local influence, GEM-Bench also introduces local ad-sensitive metrics. If an ad sentence $d_i \in D$3 is inserted between $d_i \in D$4 and $d_i \in D$5, then ad flow is

$d_i \in D$6

The score is highest when the ad is semantically balanced with respect to its left and right context. Ad coherence compares the ad sentence to the mean embedding of all non-ad sentences, thereby measuring thematic alignment between the inserted ad and the surrounding informational content (Hu et al., 17 Sep 2025).

Two additional operational metrics are Injection Rate and Click-Through Rate. Injection Rate is binary at the response level and measures whether at least one ad is inserted; aggregated over the dataset, it becomes the fraction of responses containing an ad. CTR is treated in the standard way as the ratio of clicks to impressions, although in GEM-Bench it is simulated rather than observed from live users. The benchmark notes that IR functions as an upper bound on the probability that a user can notice an ad (Hu et al., 17 Sep 2025).

Qualitative evaluation is performed with LLM-as-a-Judge. For each of the six high-level metrics, the judge scores two associated dimensions as bad, moderate, or good, and these paired judgments are mapped to $d_i \in D$7. The qualitative rubric is grounded in a codebook derived from 179 human participants and includes themes such as “jump-out” feeling, helpfulness, persuasiveness, credibility, and perceived sponsor bias. Multiple judge models are used—gpt-4.1-mini, qwen-max, claude-3-5-haiku, and kimi-k2—to reduce judge-specific alignment effects (Hu et al., 17 Sep 2025).

This evaluation design makes GEM-Bench structurally different from reference-matching setups such as BLEU- or ROUGE-centered NLG benchmarks. The AIR task is evaluated as a combination of semantic flow, thematic fit, promotional visibility, perceived helpfulness, and predicted click behavior, rather than as similarity to a fixed reference response.

4. Baseline systems and the Ad-LLM framework

GEM-Bench includes two baseline families: the prompt-based Ad-Chat baseline and the multi-agent Ad-LLM framework (Hu et al., 17 Sep 2025).

Ad-Chat is a system-prompt approach. It first assigns the user query to one of the predefined Google Topics-style subtopics, then selects a product from the topic’s bidding products, and finally injects the ad description and URL into the system prompt so that the LLM generates a promotional response directly. Ad selection and response generation are therefore tightly coupled. This baseline does not explicitly optimize insertion position in sentence space; instead, the prompt instructs the LLM to promote the product while answering the query (Hu et al., 17 Sep 2025).

Ad-LLM decouples response generation, ad retrieval, insertion, and rewriting. Its four agents are Response-Generator, Ad-Retriever, Ad-Injector, and Response-Rewriter. The offline stage computes semantic embeddings for all ads. Online, the Response-Generator produces an initial ad-free response $d_i \in D$8; the Ad-Retriever selects a top-$d_i \in D$9 candidate set $M$0; the Ad-Injector chooses both an ad and an insertion point; and the Response-Rewriter optionally smooths the integrated response into a final AIR $M$1 (Hu et al., 17 Sep 2025).

The injection step is formulated as

$M$2

where $M$3 measures local disturbance. A simple disturbance function used in the benchmark is

$M$4

This directly encodes the local drop in flow induced by insertion. Lower disturbance means the ad is inserted where semantic continuity is least damaged (Hu et al., 17 Sep 2025).

Three Ad-LLM variants are evaluated. GI-R performs Generate and Inject with response-based retrieval, but no rewriting. GIR-R adds rewriting while keeping response-based retrieval. GIR-P also includes rewriting but uses query-based retrieval rather than response-based retrieval. These variants expose two architectural factors: whether the system should rewrite after insertion, and whether retrieval should condition on the original query or on the generated ad-free response.

The framework is explicitly extensible. Although the reported experiments fix $M$5, the paper notes that multi-ad insertion can be handled by iterative selection and injection. This suggests that GEM-Bench is designed not only as a static benchmark, but also as an experimental substrate for more sophisticated AIR policies.

5. Empirical findings and trade-offs

The main empirical result is that Ad-LLM variants outperform Ad-Chat on most user-satisfaction metrics, while Ad-Chat remains competitive or superior on some engagement metrics, especially CTR in the search setting (Hu et al., 17 Sep 2025).

On MT-Human, the best Ad-LLM instance improves the overall quantitative score by 8.4% and the overall qualitative score by 10.7% relative to Ad-Chat. On LM-Market, the corresponding gains are 1.5% and 10.4%. On CA-Prod, they are 3.8% and 8.6%. The reported qualitative improvements include gains of up to 17.6% in Accuracy, 23.3% in Personality, and 17.2% in Trust (Hu et al., 17 Sep 2025).

Injection behavior differs sharply between the two baseline families. Ad-Chat has lower IR in the chatbot datasets—63.33% on MT-Human and 95.34% on LM-Market—because the LLM may decline to include promotional content despite the prompt. By contrast, Ad-LLM variants reach approximately 100% IR whenever the query is treated as ad-suitable. This is operationally important because IR bounds the achievable ad notice rate (Hu et al., 17 Sep 2025).

CTR reveals the opposite pattern in the search setting. On CA-Prod, Ad-Chat obtains CTR $M$6, compared with 34.42–39.78 for Ad-LLM variants. The benchmark interprets this as a consequence of stronger LLM-mediated ad selection in Ad-Chat, even though the resulting responses are typically less satisfactory on accuracy, trust, and personality (Hu et al., 17 Sep 2025). This suggests that engagement-optimized prompting can be effective at surfacing clickable ads while simultaneously degrading the assistant-like character of the response.

The ablation results sharpen the architectural trade-off. GI-R, which omits rewriting, tends to score slightly better on the quantitative flow/coherence metrics because insertion is optimized directly at the sentence level. However, GI-R performs markedly worse on qualitative measures, especially Naturalness, with degradations up to 28.4% relative to rewriting variants. Rewriting therefore appears to function as a semantic repair stage: it does not merely paraphrase, but reconciles the inserted ad with discourse expectations around tone, continuity, and credibility (Hu et al., 17 Sep 2025).

Retrieval source also matters by scenario. In chatbot datasets, response-based retrieval performs better: the generated ad-free response carries richer intent information than the initial query, especially when the query is underspecified. In the search dataset, query-based retrieval performs better because the keyword-oriented query already encodes the commercial intent more directly. The benchmark also reports an embedding ablation showing that text-embedding-3-small improves overall qualitative performance over all-MiniLM-L6-v2 on the chatbot datasets, while the two are nearly identical on CA-Prod (Hu et al., 17 Sep 2025).

Cost measurements are expressed through ITTFT, OTTFT, and an aggregate

$M$7

GI-R is the lowest-cost Ad-LLM variant. Rewriting variants increase ITTFT by about an order of magnitude and roughly double OTTFT because of the extra rewrite call. Ad-Chat is particularly expensive in ITTFT due to multiple LLM-mediated topic and product selection steps and large system prompts. A plausible implication is that production AIR systems will need explicit optimization not only of quality and revenue, but also of latency and token cost (Hu et al., 17 Sep 2025).

6. Position in the benchmark landscape, limitations, and future directions

GEM-Bench occupies a different niche from earlier “GEM” benchmarks in natural language generation. The Generation, Evaluation, and Metrics benchmark and GEMv2 are living NLG benchmarks centered on datasets, metrics, multilingual coverage, and evolving evaluation practice (Gehrmann et al., 2021, Gehrmann et al., 2022). By contrast, GEM-Bench is specialized to AIR generation in Generative Engine Marketing, and its metric ontology is built around ad-specific constructs such as interruptiveness, salesmanship, notice, CTR, and perceived bias (Hu et al., 17 Sep 2025). The shared acronym therefore denotes distinct benchmark traditions.

The benchmark also differs from general LLM evaluation suites that emphasize knowledge, reasoning, or dialogue quality without modeling native advertising. In GEM-Bench, ad insertion is neither an incidental perturbation nor a post hoc stress test; it is the task itself. This suggests that AIR generation should be treated as a separate evaluation regime rather than as a variant of ordinary response generation.

Several limitations are explicit. The datasets are relatively small: MT-Human has 10 queries, LM-Market 100, and CA-Prod 120. Qualitative judgments are produced by LLM judges rather than humans, even though the qualitative codebook is grounded in human study. CTR and notice are simulated rather than measured from live users. The reported experiments assume a single inserted ad, $M$8, and focus on the AIR stage rather than the full GEM stack of bidding, auction design, and performance prediction (Hu et al., 17 Sep 2025).

The paper identifies several directions for future work. One is larger and more diverse AIR datasets, including domains where advertising raises stronger ethical or trust concerns. Another is more efficient multi-agent AIR generation, since rewriting improves quality but increases token overhead. A further direction is tighter integration between AIR generation and upstream GEM components such as auction mechanisms and CTR prediction. The benchmark’s ontology appears designed to support this extension: although introduced for AIR evaluation, it can also function as a measurement layer for broader GEM systems.

Overall, GEM-Bench establishes AIR generation as a distinct benchmarking problem: one in which relevance, discourse fit, credibility, and monetization must be measured jointly rather than optimized in isolation. That framing is its principal contribution (Hu et al., 17 Sep 2025).