LLM Retrieval for Stable and Predictable Ad Recommendations

Abstract: Traditional ads recommendation systems have primarily focused on optimizing for prediction accuracy of click or conversion events using canonical metrics such as recall or normalized discounted cumulative gain (NDCG). With the hyper-growth of ads inventory and liquidity with generative AI technologies, the prediction stability and predictability is becoming increasingly critical. Intuitively, prediction stability and predictability can be defined to quantify system robustness with respect to minor/noisy input (ads, creatives) perturbations, the lack of which could lead to advertiser perceivable problems such as repeatability, cold start and under-exploration. In this paper, we introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system, and present an online validated semantic candidate generation framework powered by fine-tuned LLMs that showed significant improvement along these metrics by fundamentally improving the semantic-awareness of the system. The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the foundation for graph-based expansion, ensuring the retrieved candidates encapsulate semantic variants of an ad, guaranteeing that small creative variants from the advertiser yield consistent and explainable delivery results to the user. We tested this LLM ads retrieval framework in a large-scale industrial ads recommendation system, demonstrating significant improvements across offline and online A/B experiments, showcasing gains in both predictability and traditional performance metrics. Although evaluated in the ads stack, this is a general framework that can be applied broadly to any large-scale recommendation and retrieval systems facing similar scaling and predictability challenges.

• The paper demonstrates that LLM-driven semantic ad retrieval improves predictability by reducing the A/A' difference by 8.62%.
• It introduces a novel framework that integrates LLMs with graph-based semantic candidate generation to enhance recall and delivery consistency.
• Experimental evaluations reveal a 0.45% increase in conversions and a 45% reduction in impression volatility, validating the approach.

LLM-Based Retrieval for Predictable and Stable Ad Recommendation

Introduction

Advancements in ad recommendation systems have traditionally prioritized canonical objectives such as CTR maximization and recall. The proliferation of ads inventory, driven in part by generative AI, has intensified requirements for system predictability and stability. The study "LLM Retrieval for Stable and Predictable Ad Recommendations" (2605.21969) introduces a novel framework that explicitly quantifies and optimizes for system predictability in ad retrieval. Leveraging fine-tuned LLMs, the work proposes a semantic candidate generation pipeline to deliver robust, context-aware ad recommendations that align with both user and advertiser expectations.

Metrics for Predictability and Performance

The authors formalize the notion of predictability by introducing the A/A' difference metric. This metric quantifies delivery consistency when input ads experience minor, semantically neutral perturbations—a critical consideration for recurrent ad uploads, cold starts, and dynamic creative changes. Predictability is operationalized as the statistical significance difference (StatSigDiff) between performance metrics (such as conversions) of primary and shadow ad pairs. The derivation normalizes for inherent stochasticity and revenue impact, providing a domain-aligned stability guardrail for model development.

Complementary metrics include Recall@K, median absolute deviation (MAD) of impression differences, and online top-line metrics (e.g., conversions, clicks). Collectively, they enable comprehensive evaluation of both precision/recall trade-offs and stability improvements introduced by LLM-based retrieval.

Semantic Candidate Generation with LLMs

Traditional candidate generators for ad retrieval—often based on shallow features, embeddings, or graph heuristics—struggle with semantic equivalence and susceptibility to non-semantic perturbations. The presented framework instead capitalizes on pre-trained and fine-tuned LLMs to extract hierarchical and semantically rich representations from ad creatives.

The system fine-tunes an LLM on ad engagement data, learning to encode ads into discrete semantic tokens and categorical attributes. These representations are then structured into a graph, where nodes are ads and edges capture shared semantic attributes. Graph traversal algorithms, leveraging set-based similarity (e.g., Jaccard index over phrases/tokens), enable robust candidate expansion and recall maximization.

Figure 1: LLM Ad to Ad Generator—pipeline for extracting semantic ad features and building candidate pools via LLM-powered attribute and category generation.

Figure 2: LLM Contextual Category Graph—illustrates the graph construction with ads clustered according to LLM-extracted semantic categories and attributes.

This hierarchical clustering inherently reduces system sensitivity to superficial creative changes—ensuring that ad delivery is consistent across semantically equivalent variants. The real-time retrieval layer is implemented over distributed GPU infrastructure to accommodate industrial-scale latency and throughput requirements.

Experimental Evaluation

Evaluation was conducted through large-scale A/B testing using Meta's industrial ad delivery stack. The LLM-based retrieval model (LLama3-8B Instruct, zero-shot mode) replaced the baseline ensemble candidate generator, benchmarking both traditional performance metrics and the proposed predictability metrics.

Key results include:

• Topline performance: Statistically significant 0.45% increase in primary online metrics (e.g., conversions) versus control.
• Recall: Final stage recall improved by 1.2%, validating the hypothesis that LLM-driven semantic expansion enhances relevant candidate coverage.
• Quality-aware ranking: The Recall Alignment Ratio decayed with increasing K, indicating the LLM's ability to prioritize top-quality candidates.
• Predictability: StatSigDiff (A/A' difference) exhibited a statistically significant 8.62% relative reduction, signifying superior delivery consistency under input perturbations.
• Variance reduction: Median Absolute Deviation for impression difference between primary and shadow ad pairs improved by 45%, reflecting substantially lower volatility.

  Figure 3: Daily impression relative difference between the A/A' ad pairs for the test vs control—demonstrates reduced volatility and improved predictability delivered by the LLM system.

These results substantiate the claim that LLM-powered semantic candidate generation not only improves retrieval efficacy but also promotes stability and explainability—two properties underrepresented in prior work.

Implications and Future Directions

The integration of LLM-driven semantic retrieval into ad recommendation systems represents a paradigm shift toward robust, explainable, and advertiser-aligned optimization objectives. By introducing formal predictability metrics and structural invariance to semantically neutral perturbations, the solution addresses practical concerns of repeatability, cold start inefficiency, and under-exploration in dynamic ad ecosystems.

This architecture is architecturally modular, suggesting broad applicability to other large-scale retrieval tasks—such as content or product recommendation—where predictability is subject to similar challenges. Realizing full-stack, semantically aware recommender systems with adaptive multimodal learning and real-time generalization (e.g., for unseen creative modalities or cross-domain transfer) is identified as a critical avenue for future work.

Conclusion

The study demonstrates that leveraging fine-tuned LLMs for semantic ad representation and candidate retrieval improves both canonical ad performance and critical predictability metrics, as validated in a rigorous industrial-scale A/B test. The explicit incorporation of predictability as an optimization objective not only enhances system robustness but also aligns ad delivery systems with advertiser and user expectations in highly dynamic inventory environments. Expanding these capabilities toward end-to-end, multimodal, and continual learning systems constitutes a compelling direction for future AI-powered retrieval and recommendation research.