An LLM-powered Agentic Recommendation System for Connected TV Content Discovery

Abstract: Recommendation systems, from traditional multi-stage to recent unified generative architectures, face challenges in incorporating diverse contextual signals, such as trending topics, breaking news, cultural events, and cross-surface user activities, into their ranking pipelines. These systems are designed to consume structured behavioral signals with consistent schemas, and lack the reasoning capability to naturally process unstructured or heterogeneously formatted contextual information. Incorporating such signals typically requires feature engineering, bespoke data pipelines, and carefully tuned heuristics. In this paper, we present an LLM-powered agentic recommendation system designed for Connected TV (CTV) content discovery that addresses these limitations. Our system leverages the reasoning capabilities of LLMs to naturally process and synthesize diverse signals across varying schemas and structures, eliminating much of the manual integration inherent in traditional ranking and retrieval systems. Recognizing that current LLM-based solutions still fall short of traditional machine learning models in several recommendation tasks, including retrieval efficiency, personalization precision, and scalability, we adopt an agentic architecture that orchestrates specialized components, allowing each sub-task to be handled by the most suitable method, whether LLM-based or traditional ML. The main contribution of this work is our engineering approach to successfully overcoming the practical limitations of enabling LLM for recommendation, particularly inference latency. We share insights from our work and discuss the trade-offs and lessons learned in building a hybrid system that combines the flexibility of LLMs with the performance of established recommendation techniques.

• The paper introduces an agent-based recommendation system that leverages LLMs to synthesize heterogeneous, real-time contextual data for CTV content.
• It integrates natural language processing with traditional ranking models, using prompt engineering and generative retrieval via MediaBrain.
• Empirical results demonstrate improved recommendation adaptability and reduced latency through asynchronous inference and beam search optimization.

Agentic LLM-powered Recommendation for Connected TV Content Discovery

Motivation and Problem Setting

Contemporary industrial recommendation frameworks, whether multi-stage stacks or unified generative architectures, are fundamentally limited in integrating heterogeneous and unstructured contextual signals—such as trending events, breaking news, seasonal phenomena, and cross-surface user interaction—without major engineering overhead and latency trade-offs. This constraint is accentuated in the Connected TV (CTV) context, where the discovery interface requires semantically organized multi-topic carousels, real-time adaptation to context, and personalization that fuses editorial curation with user preferences.

The paper presents an agentic recommendation architecture leveraging LLMs for autonomous reasoning and content orchestration within Meta’s CTV platform. The system transitions from static, retrain-heavy, manual-feature pipelines to an orchestrated, agentic framework wherein promptable LLM agents handle open-ended contextual ingestion, topic generation, and ranking. Legacy high-performance ML ranking models are retained in latency-critical stages for optimal engagement metrics.

System Architecture

The architecture is structured as an orchestrated multi-agent pipeline, each agent assigned to a well-scoped functional domain and interfacing via a shared context layer composed of natural-language features. The orchestrator aggregates and serializes multi-source context—encompassing both structured user behavior signals and unstructured realtime data—into inputs for downstream agents.

Figure 1: The orchestrator coordinates an LLM-driven topic retrieval agent, a hybrid media retrieval and ranking agent (with MediaBrain as the generative retrieval core), and a topic ranking agent for multi-carousel content layout.

The pipeline proceeds as follows:

• LLM Topic Retrieval Agent: Consumes rich, natural-language context (composed with current-time, trending events, user interests, past session behaviors) to propose topic slates for potential home screen channels using an instruction-tuned Llama-3 70B. It filters outputs via curated blocklists and classifiers.
• Media Retrieval and Ranking Agent: For each candidate topic, hybrid retrieval obtains and ranks media items. The MediaBrain generative retrieval model (see below), supported by classic MTML models for personalization, maps topic prompts to semantic media candidates (SIDs) and handles media–topic alignment. This allows direct, prompt-driven signal injection and bypasses fixed-vocabulary constraints.
• LLM Topic Ranking Agent: Integrates per-topic engagement, inventory stats, and media quality, issuing a final, context-sensitive topic ordering for display.

Generative Retrieval: MediaBrain

MediaBrain is a central innovation: a fine-tuned Llama 3.2 1B LLM that translates natural language topic prompts into quantized semantic identifiers (SID tokens) optimized through residual k-means quantization. The model undergoes supervised alignment training over curated topic–media pairs and is evaluated both at head (CTV100/1000) and long-tail (niche entity) topic levels.

Key aspects:

• Inference Pipeline: Autoregressive token generation (beam search, typically ~500 ms) followed by a prefix lookup over pre-indexed SID–media mappings.
• Prompt Adaptation: The model’s generative flexibility enables utility beyond static training distributions; context-specific, style-aware, and platform-tuned instructions can refine retrieval without retraining, as empirical prompt interventions demonstrate.

Figure 2: Media retrieval relevance for MediaBrain across iterations—substantial improvement for previously weak, niche-entity retrievals, with scores rising from 0.15 to 0.50 in hard cases.

Empirical analysis indicates the system’s major limiting factor for niche and entity-centric topics is the granularity of the SID tokenization and the embedding space’s representational bottleneck.

Engineering for Production: Latency and Throughput Optimizations

A critical deployment concern for CTV is interactive latency (sub-500 ms SLA) and scalability at industrial QPS. Key engineering strategies include:

• Cache-based Asynchronous Inference: All LLM operations are shifted off the UI request path, with agent outputs cached per component; cache lifecycles are fine-tuned to align with context volatility (e.g., long-term interests cached more aggressively, in-session signals fetched fresh).
• Beam Search Optimization: Multi-tiered serving tweaks—moving to concurrent request handling, addressing serialization and GIL bottlenecks, and distributing beam search orchestration—push throughput from 2 QPS to 200 QPS per GPU, a cumulative 100× improvement.
• Session Cursor Tokenization: Generated SID sets are persisted and propagated across session “chaining pages” to avoid redundant generation, bounding compute costs to one per session rather than per request.
• Context and Feedback Logging: All agent boundary outputs are logged, enabling provenance for debugging and potential future feedback integration.

Empirical Results

Offline LLM-judge-based evaluations characterize retrieval quality across topic popularity strata. Important findings include:

• Training Data Quality: Surface-level alignment of media content and topic prompts is foundational; filtered and balanced datasets produce higher signal-to-noise for SFT, directly affecting relevance.
• Prompt-context Injection: Generative retrieval behavior can be robustly adjusted to contextually nuanced or client-specific styles in production through prompt engineering alone, evidencing high instruction-following capacity post-SID alignment.
• SID Topic Dispersion and HHI: Topic “hardness” (in retrieval and relevance) positively correlates with SID dispersion, as measured by Herfindahl-Hirschman Index. Highly dispersed niche topics yield lower relevance, indicating an expressivity limit in current quantization/embedding.

Figure 3: L1 SID distribution illustrates the contrast between concentrated topics (tightly aligned content, e.g., Basketball) and dispersed, ambiguous topics (e.g., Hello Kitty), the latter suffering lower relevance.

Discussion, Limitations, and Future Directions

Agentic Feedback Loops

The present system is a unidirectional pipeline with no upstream propagation of downstream quality or engagement signals. Incorporating closed-loop memory—cross-session persistent traces of what fails (e.g., topics with perennially poor media coverage, skipped categories)—could enable continual agentic refinement with no model retraining.

SID Bottleneck and Niche Topic Ceiling

Despite data augmentation, generative retrieval for niche topics saturates early, constrained by the granularity of the SID token space. This indicates an architectural trade-off between quantization efficiency and topical specificity—a bottleneck that future work aspires to track using the HHI as a lightweight proxy for “learnability” diagnostics.

Practical Implications

The key theoretical implication is that LLM-enabled agents can operationalize a form of open-domain bridging for recommendation, digesting and acting on rich, real-world context without static-engineering bottlenecks. Practically, this shifts the maintenance burden from feature creation and retraining to context and prompt orchestration, enabling rapid, low-cost adaptation to changing domain requirements and new signal streams.

The system achieves industrial-grade latency at Meta scale by relegating LLM inference to the asynchronous, cache-warmed path and retaining fast classical models for personalization in latency-critical stages.

Conclusion

This work demonstrates the viability and value proposition of agentic, context-driven LLM-powered recommendation systems for Connected TV, combining prompt-driven reasoning and the strengths of classical ranking systems. By orchestrating these hybrid approaches, the system flexibly digests unstructured context, dynamically injects real-world events into recommendations, and achieves operational latency and throughput suitable for production.

Empirical analysis identifies both the capacity and limits of generative retrieval: while prompt/agent-based architectures enable rapid adaptation and cross-domain signal inclusion, bottlenecks in fine-grained media representation (SID dispersion) persist, especially for niche entities. The proposed path forward leverages contextual feedback memory and continuous SID schema diagnostics, suggesting an evolving research frontier for agentic, context-aware recommendation engines at scale.