LLM Ads Retrieval Framework

• LLM Ads Retrieval Framework is defined by its use of LLM-derived semantic representations, replacing traditional keyword-based ad retrieval with semantic indexing.
• It encompasses diverse architectures such as RARE and LEADRE, which integrate direct query-to-ad pathways and retrieval-augmented generation for optimized candidate selection.
• Empirical evaluations report significant improvements in metrics like CTR, GMV, and recall, demonstrating scalable deployment in high-volume, real-world advertising systems.

An LLM ads retrieval framework is a family of advertising systems in which a LLM participates directly in retrieval, candidate generation, semantic indexing, or retrieval-augmented ad insertion, rather than being confined to downstream generation or generic feature extraction. In the current literature, this design appears in sponsored search, display advertising, advertiser keyphrase recommendation, advertiser prediction, and generative chatbot monetization. Its shared premise is that LLM-derived semantic structure can replace or augment ID-based, keyword-only, or embedding-only retrieval while remaining compatible with large-scale serving constraints (Liu et al., 2 Apr 2025, Li et al., 2024, Dey et al., 5 Aug 2025, Hajiaghayi et al., 2024).

1. Core architectural patterns

A canonical industrial instance is RARE, short for “Real-time Ad REtrieval.” RARE replaces the classic query→keyword→ads cascade with a direct query→Commercial Intent (CI)→ads pathway. Its online path comprises four numbered modules: a Query Analyzer & Cache, a CI Generator implemented as a small 1 B-parameter LLM with constrained beam search, a CI→Ads Retrieval module backed by an inverted index, and a downstream Ad Scoring & Ranking stage. The cache covers ~5 % of queries but ~60 % of traffic; misses are sent to the online LLM, which operates under a latency budget of ≤ 60 ms. The retrieved ads are the union of the buckets associated with the generated CIs, and the top-$K$ ads are returned to serving (Liu et al., 2 Apr 2025).

LEADRE, short for “LLM Empowered Display ADvertisement REcommender system,” uses a different retrieval architecture. It is inserted as a retrieval sub-branch alongside traditional ID-based recall inside a conventional display-ad funnel consisting of retrieval, coarse ranking, final ranking, and re-ranking. Traditional recall uses a two-tower model to pull ∼1k candidates, while LEADRE’s generative recall calls a fine-tuned LLM to generate the next-ad S-ID under a trie-tree constraint and produces its own top-$B$ list with $B\approx50$–100; the two candidate sets are then merged before ranking (Li et al., 2024).

Other frameworks generalize the same principle without using CI or S-ID as the intermediate object. “LLM Retrieval for Stable and Predictable Ad Recommendations” extracts hierarchical semantic attributes from ad creatives, then uses those representations for graph-based candidate expansion. “Fine-Tuned LLM as a Complementary Predictor Improving Ads System” uses a fine-tuned open-source LLM as an ads-specific ancillary predictor, forecasting likely advertisers from user profiles and histories, and adds a new retrieval channel on top of an existing candidate-generation stack. “LLMDistill4Ads” decomposes retrieval into a teacher–assistant–student pipeline in which an LLM judge labels item–keyphrase pairs, a cross-encoder calibrates those judgments, and a bi-encoder serves the final retrieval model (Sunkara et al., 21 May 2026, Yang et al., 27 May 2026, Dey et al., 5 Aug 2025).

A recurrent misconception is that LLM-based ad retrieval is synonymous with direct embedding similarity. The literature shows several distinct retrieval substrates: generated Commercial Intent texts, generated semantic-IDs, hierarchical semantic attributes, cross-encoder-to-bi-encoder distillation, advertiser-list prediction, and retrieve-then-generate auction mechanisms (Liu et al., 2 Apr 2025, Li et al., 2024, Sun et al., 15 May 2026).

2. Semantic representations and learning objectives

The central design choice in these systems is the intermediate semantic representation. In RARE, the intermediate object is the Commercial Intention. The formal setup defines $Q$ as the space of user queries, $\mathcal{C}$ as the space of all possible Commercial Intent texts, and $A$ as the ad corpus. CI generation is modeled as a learned mapping from query to a subset of $\mathcal{C}$, and retrieval over multiple generated CIs is written as

$$A_{\mathrm{CI}(q)} \;=\;\bigcup_{c\in\mathrm{CI}(q)} A_c.$$

Each CI corresponds to multiple ads, yielding a lightweight and scalable set of CIs, in contrast to one-to-few numeric DocID mappings (Liu et al., 2 Apr 2025).

LEADRE uses semantic-IDs rather than free-form CI text. Its fine-tuning prompts combine a task instruction, user profile, long-term interest summary, short-term ad-domain behavior, and short-term content-domain behavior, and the model is trained with next-token cross-entropy over S-ID sequences. To bridge the knowledge gap between LLMs and advertising semantics, LEADRE adds two auxiliary alignment tasks—Explicit Alignment and Implicit Alignment—and then applies Direct Preference Optimization on triplets $(u,a_h,a_l)$ where $ECPM(a_h) > ECPM(a_l)$. The reported production configuration uses Hunyuan 1B with RQ-VAE S-IDs of length 5, 4 quantization steps, and 1024 codes/layer (Li et al., 2024).

The stable-and-predictable framework represents each ad by hierarchical category IDs $B$0 and attribute–score pairs $B$1. The ad representation is organized across semantic levels from broad to fine, and the framework maps each attribute to a dense embedding $B$2. This hierarchy allows retrieval to operate at coarse category or fine attribute levels depending on infrastructure capacity. A plausible implication is that this representation is designed not only for recall, but also for robustness under semantically neutral perturbations of creative assets (Sunkara et al., 21 May 2026).

LLMDistill4Ads adopts a different supervision strategy because negative human judgment is difficult to procure at scale. It uses Mixtral 8×7B as an LLM judge to produce yes/no relevance labels for item–keyphrase pairs, trains a microBERT cross-encoder on those labels with binary cross-entropy, and then trains a bi-encoder student with a weighted multi-task loss combining multiple-negatives ranking on CTR data, contrastive loss on Search-Relevance and LLM-labeled data, and Pearson rank-imitation from the cross-encoder logits: $B$3 The same paper reports that pointwise MSE, neural‐nDCG and LambdaLoss were tried and discarded (Dey et al., 5 Aug 2025).

AdNanny extends the supervision regime further by constructing reasoning-augmented corpora of the form $B$4, then performing multi-task supervised fine-tuning with adaptive instance- and task-level reweighting, followed by KL-regularized GRPO against offline retrieval and ranking evaluators. This places offline ad retrieval inside a unified reasoning-centric backbone rather than a standalone retrieval model (Hu et al., 2 Feb 2026).

3. Retrieval operators, indexes, and candidate-generation mechanisms

The retrieval operator in RARE is an inverted CI→ads index built offline. Index construction generates CIs for every ad with a larger LLM of 13 B parameters and beam size 256, cleans and clusters the outputs into $B$5 with $B$6, and maintains a hash map from each $B$7 to its ad list. New ads receive ~30 CIs per ad through constrained beam-search decoding, the full CI set is re-refreshed monthly, and ad-level inserts happen hourly. The stated complexity is build cost $B$8 decoding steps and lookup cost $B$9 hash lookups, followed by linear union of buckets (Liu et al., 2 Apr 2025).

The stable-and-predictable framework uses a semantic graph rather than an inverted text index. Its node types are ad nodes and attribute nodes. Ad–attribute edges are weighted by LLM attribute scores, while ad–ad edges are induced via Jaccard similarity over phrase sets or token sets. Candidate expansion can be performed by nearest-neighbor retrieval,

$B\approx50$0

or by multi-hop random-walk expansion, which is intended to surface both highly precise matches and broader semantic variants (Sunkara et al., 21 May 2026).

LLMDistill4Ads serves retrieval through precomputed 64-d Matryoshka embeddings and HNSW-based ANN, with optional re-ranking by the cross-encoder. The deployment path includes de-duplication against existing recall models and a production relevance filter. In the complementary-predictor framework, the LLM produces a top-20 advertiser set $B\approx50$1, after which a standard two-tower ANN search is run only over creatives whose advertiser lies in that set. The combined candidate-generation score is

$B\approx50$2

with $B\approx50$3–1.0 and $B\approx50$4 controlling the LLM-channel quota (Dey et al., 5 Aug 2025, Yang et al., 27 May 2026).

LEADRE’s retrieval operator is constrained generation over a trie-tree of semantic-IDs rather than ANN over embeddings. This enables a generative branch to emit valid ad identifiers directly, then merge them with a two-tower recall branch. MarketingFM, by contrast, uses a retrieval-augmented generation module for marketing copy: sentence-transformer embeddings are indexed with FAISS over two product knowledge bases, and the retriever returns top-$B\approx50$5 products as prompt context. This suggests that, within the broader LLM advertising literature, “retrieval framework” includes both ad-candidate retrieval and product-context retrieval for downstream ad creation (Li et al., 2024, Liu et al., 22 Jun 2025).

4. Evaluation regimes and reported empirical outcomes

RARE reports both offline and online evaluation. Offline, the dataset contains 5 000 head queries with up to 1 000 ground-truth clicked ads each and a total ad pool of ≈ 150 000. The metrics are Ad Coverage Rate (ACR), Hit Ratio@K, and MAP. At $B\approx50$6, RARE reports $B\approx50$7, $B\approx50$8, and $B\approx50$9, while the best non-RARE baseline reports $Q$0 and $Q$1. Online, a 20 % traffic slice for 1 month on Tencent platforms yields, on WeChat Search, Consumption ↑ 5.04 %, Gross Merchandise Volume (GMV) ↑ 6.37 %, CTR ↑ 1.28 %, Shallow Conversions ↑ 5.29 %, and Deep Conversions ↑ 24.77 %; additional gains are reported on DSP and QQ Browser (Liu et al., 2 Apr 2025).

The stability-and-predictability framework introduces new system-level metrics in addition to Recall@K and NDCG@K. Predictability is measured by StatSigDiff over primary–shadow ad pairs, and stability is measured by the median absolute deviation (MAD) of daily relative impression differences. In a 14-day online A/B test with a 50% control and 50% treatment split over tens of millions of ads, the paper reports an online top-line ads performance lift of +0.45% ($Q$2), final-stage recall +1.2% ($Q$3), Predictability (StatSigDiff) –8.62% relative reduction vs. control, and Stability (MAD of daily impressions) –45% (Sunkara et al., 21 May 2026).

LLMDistill4Ads evaluates both offline retrieval quality and business metrics. The cross-encoder assistant reaches F1 91 %, P 92 %, R 90 % on 7.5 M held-out LLM-labeled pairs. In an online A/B test in the U.S. over 12 days, the reported effects are Clicks +11.5 % ($Q$4), Gross Merchandise Volume (GMB) +51.26 % ($Q$5), and ROAS +38.69 % ($Q$6). The paper also reports that the Pearson distillation loss yields CE-corr 0.79, F1 89 %, P 87 %, R 91 %, outperforming MSE and CoSENT under the reported setup (Dey et al., 5 Aug 2025).

The complementary-predictor framework evaluates advertiser prediction and its impact on downstream conversion models. Its best advertiser-prediction setting, SID-enabled SFT (1-advertiser prompt)+GRPO, reports Recall@1 $Q$7, Recall@5 $Q$8, and Recall@20 $Q$9. The addition of LLM-derived features produces offline conversion-model changes of $\mathcal{C}$0Loss –0.12% and $\mathcal{C}$1PR-AUC +0.71% for ctcvr, and $\mathcal{C}$2Loss –0.64% and $\mathcal{C}$3PR-AUC +1.64% for vtcvr. Online, U.S. Shopping Ads RoAS gains are +4.94 % ($\mathcal{C}$4) on Full U.S. Shopping and +6.69 % ($\mathcal{C}$5) on Opt-in U.S. Shopping (Yang et al., 27 May 2026).

LEADRE reports offline next-ad prediction metrics and online GMV. In the full ablation table, LEADRE reaches HR@1 $\mathcal{C}$6, HR@4 $\mathcal{C}$7, HR@8 $\mathcal{C}$8, NDCG@4 $\mathcal{C}$9, and NDCG@8 $A$0; the EX→IM→Main tuning order improves further to HR@4 $A$1 and NDCG@4 $A$2. Online A/B testing on Tencent WeChat Channels and Moments with a 20% traffic split reports +1.57% GMV lift on Channels ($A$3) and +1.17% GMV on Moments ($A$4). AdNanny, in Bing Ads, reports that a bi-encoder trained on AdNanny-labeled relevance data reaches Recall@10 57.2 %, Recall@50 87.0 %, and MRR 0.607, versus 47.3 %, 78.4 %, and 0.527 for the human-label baseline (Li et al., 2024, Hu et al., 2 Feb 2026).

5. Retrieval-augmented generation and auction-based ad insertion

A distinct branch of the literature studies ad retrieval inside the outputs of LLMs themselves. “Ad Auctions for LLMs via Retrieval Augmented Generation” divides the final output into $A$5 discourse segments and, for each segment $A$6, computes ad relevance probabilities $A$7. In the single-ad mechanism, each advertiser submits a bid $A$8, and the winning probability becomes

$A$9

The paper derives an incentive-compatible pricing rule, proves dominant-strategy incentive compatibility and individual rationality, and shows that the mechanism maximizes logarithmic social welfare. It also gives a multi-ad extension and reports a revenue/quality trade-off: more frequent single-ad insertions maximize revenue and welfare, whereas bulk multi-ad insertions improve output quality as measured by cosine-similarity between ad-free and ad-conditioned outputs (Hajiaghayi et al., 2024).

“Mechanism Design for Quality-Preserving LLM Advertising” introduces an explicit organic-content baseline into the welfare definition. At each generation step, the retrieval set is $\mathcal{C}$0, and the mechanism derives an endogenous reserve price

$\mathcal{C}$1

from marginal social welfare, screening out ads with $\mathcal{C}$2. The paper develops a KL-regularized single-allocation mechanism with Myerson payments and a screened VCG multi-allocation mechanism, both satisfying dominant-strategy incentive compatibility and individual rationality. Over four synthetic and semi-synthetic scenarios, revenue per ad climbs by 30–50% under the quality-preserving single-allocation mechanism, semantic relevance and social welfare rise by 10–30%, KL divergence to the purely organic RAG model is driven close to zero ($\mathcal{C}$3–0.05 versus 0.15–0.3 for baselines), and human-proxy semantic-similarity scores remain within 2–5% of the no-ad outputs (Han et al., 7 May 2026).

LERA, or “LLM-Enhanced RAG for Ad Auction in Generative Chatbots,” refines the retrieve-then-generate pattern through two stages. Stage 1 performs embedding-based coarse filtering with sentence-transformers/all-MiniLM-L6-v2 and chooses top-$\mathcal{C}$4 advertisers by a bid-weighted similarity score. Stage 2 asks the LLM itself to choose exactly one candidate or “No insertion,” extracts sequence-level logits, converts them into normalized organic relevance scores,

$\mathcal{C}$5

and then applies a critical-value payment rule that accounts for both the Stage 1 entry threshold and the Stage 2 win threshold. On a synthetic advertiser-query benchmark, LERA reports ≈94.6% single-ad selection accuracy with a 4 B model and ≈98.7% with a 30 B model, versus ≈63% for the embedding-only baseline; for multi-ad query satisfaction, LERA (30 B) reports ≈80% average coverage of requested categories (Sun et al., 15 May 2026).

6. Deployment constraints, limitations, and open directions

The deployment record of these systems shows that LLM-based ad retrieval is not restricted to laboratory settings. RARE has been implemented in a real-world online system handling daily search volumes in the hundreds of millions; its head-query cache hit rate is ~65 %, reducing online LLM calls by ~70 %, while the online 1 B model is FP8 quantized and runs on hundreds of NVIDIA L40 GPUs at peak GPU utilization ~90 %. LEADRE has been deployed on WeChat Channels and Moments, serving tens of billions of requests each day, using a hybrid latency-sensitive and latency-tolerant service design. LLMDistill4Ads combines full-batch inference over 2.3 B items with near-real-time updates through Flink, Triton/ONNX, and daily ANN refresh. AdNanny relies on 248 GPUs across 31 machines, hybrid dense–MoE parallelism, and FP8 inference for offline production use in Bing Ads (Liu et al., 2 Apr 2025, Li et al., 2024, Dey et al., 5 Aug 2025, Hu et al., 2 Feb 2026).

The limitations reported across papers are structurally consistent. RARE states that current CI generation and retrieval are decoupled, so the model does not jointly discriminate relevance while generating; future work aims to integrate a discriminative head into the LLM so that it can assign relevance scores $\mathcal{C}$6 during generation. The stability-and-predictability framework highlights infrastructure cost for LLM inference, graph freshness, multimodal extension, online adaptive learning of graph weights, and formal optimization of the Stability–Recall trade-off. LLMDistill4Ads notes that hard negative mining was not integrated due to scale constraints, and that direct LLM softmax proved too spiky compared with cross-encoder calibration. LEADRE identifies next-$\mathcal{C}$7 generation and information-balanced S-IDs as open problems. MarketingFM, although focused on ad-copy generation rather than ad-candidate retrieval, states that human oversight remains essential for setting thresholds and validating refinements before deployment (Liu et al., 2 Apr 2025, Sunkara et al., 21 May 2026, Dey et al., 5 Aug 2025, Li et al., 2024, Liu et al., 22 Jun 2025).

This suggests that the modern LLM ads retrieval framework is no longer evaluated solely by Recall@K, NDCG, HR@K, or MAP. The literature increasingly treats semantic fidelity, predictability, stability, calibration, latency, auction-theoretic truthfulness, and business metrics such as GMV, ROAS, and consumption as co-equal design targets. In that sense, the field has expanded from “retrieval quality” in the narrow ranking sense to a broader systems objective in which retrieval must be semantically expressive, operationally scalable, and compatible with advertiser-facing and user-facing constraints (Sunkara et al., 21 May 2026, Han et al., 7 May 2026).