Large Transformer Recommenders

• Large Transformer Recommenders are deep learning models that use self-attention to capture long-range dependencies in user–item interactions.
• They employ autoregressive pretraining with dual objectives and negative sampling to efficiently scale to billion-parameter regimes.
• Two-tower architecture conversion enables rapid offline inference and effective integration into production recommendation systems.

Large Transformer Recommenders are deep learning models employing transformer architectures at significant scale to address recommendation tasks over massive user bases, item catalogs, and interaction histories. Compared to classical collaborative filtering and shallow neural approaches, these systems leverage advances in sequence modeling and representation learning to capture complex, long-range dependencies within user–item interactions. Recent research has focused on scaling transformers in recommendation contexts, optimizing their efficiency, and demonstrating their effectiveness in real-world industrial deployments.

1. Architectural Foundations and Scaling Approaches

Large Transformer Recommenders utilize transformer encoders as their main sequence modeling module, where user interaction histories (often combined with context such as device, location, or content metadata) are processed as ordered sequences. The canonical transformer layer, based on self-attention, is adept at modeling both short- and long-term dependencies and context-sensitive recommendations. The input is typically a structured sequence of triplets or tuples (context, item, feedback), with the transformer operating autoregressively to model user behavior trajectories (Khrylchenko et al., 21 Jul 2025).

Scaling transformers for recommenders introduces several domain-specific challenges:

• Parameter Growth: With increasing model size, architectures approach hundreds of millions to over a billion parameters, requiring careful design to maintain feasible training/inference cost, stability, and memory usage (Khrylchenko et al., 21 Jul 2025).
• Input Representation: User–item interactions are encoded with context (cₜ), item identity (iₜ), and observed feedback (fₜ), forming input sequences such as (c₀, i₀, f₀), (c₁, i₁, f₁),… for pretraining and fine-tuning (Khrylchenko et al., 21 Jul 2025).
• Autoregressive Pretraining: Recommendation is formulated as a sequential transduction task: the transformer learns to predict both the “impressed” item and the user feedback at each timestep, mirroring production log policies and actual engagement. The principal losses are:

$$\mathcal{L}_\text{NIP} = -\log\left(\frac{\exp(f(\hat{h}_t^c, i_t))}{\exp(f(\hat{h}_t^c, i_t)) + \sum_{n \in N} \exp(f(\hat{h}_t^c, n) - \log Q(n))}\right)$$

$$\mathcal{L}_\text{FP} = \sum_{k=1}^K \text{CrossEntropy}(f_t^k, l_t^k)$$

where $f$ is typically cosine similarity, $Q(n)$ is the negative sampling distribution, and $K$ is the number of feedback dimensions (Khrylchenko et al., 21 Jul 2025).

• Fine-Tuning: For deployment efficiency, the pre-trained encoder is converted into a two-tower model for fast candidate scoring: one tower encodes user state offline, the other encodes candidate items, and their dot product yields ranking features for large-scale inference (Khrylchenko et al., 21 Jul 2025).

The architecture blends the expressivity of transformers with practical scalability constraints, leveraging modern hardware acceleration and large-batch training regimens.

2. Training Strategies, Dual Objectives, and Negative Sampling

Large-scale transformer recommenders must address sample efficiency and unbiased learning under industrial data regimes:

• Dual-Objective Learning: By simultaneously optimizing next-item prediction (NIP) and feedback prediction (FP), the model disentangles mimicking production logging (imitating what users see) and true preference modeling (feedback). The overall loss becomes:

$$\mathcal{L}_{\text{pretrain}} = \mathcal{L}_\text{NIP} + \mathcal{L}_\text{FP}$$

This dual supervision enables the model to benefit from sequential context as well as direct behavioral signals (Khrylchenko et al., 21 Jul 2025).

• Sampled Softmax and Negative Sampling: Handling immense catalogues necessitates efficient sampled softmax with importance corrections:

$f(\hat{h}_t^c, i_t) = \frac{\cos(\hat{h}_t^c, i_t)}{e^{\tau}} \qquad \text{(with $\tau$ as trainable temperature)}$

Negative candidates $N$ are chosen via mixed strategies, with logQ correction for sampling bias. This approach balances computational tractability with effective ranking (Khrylchenko et al., 21 Jul 2025).

• Feedback Factorization: When user engagement is multidimensional (e.g., skips, likes, durations), feedback distributions are factorized and trained with cross-entropy losses per feedback type, capturing richer user signals and improving generalization (Khrylchenko et al., 21 Jul 2025).

3. System Optimization and Efficient Deployment

Enabling large-scale transformer recommenders in production environments requires addressing efficiency in both modeling and serving:

• Sequence Compression: To avoid sequence-length explosion (triplets for every timestep), a simplification is introduced: context, item, and feedback are merged into a single embedding per timestep using MLPs, reducing memory and compute overhead while preserving context (Khrylchenko et al., 21 Jul 2025).
• Parameterization and Model Size: The model family is scaled from 3.2 million to over 1 billion parameters. Efficiency is maintained through careful architectural design (e.g., removing unnecessary deep feedforward expansions; shallow projections for context fusion), and hyperparameter tuning for stability (e.g., temperature clipping) (Khrylchenko et al., 21 Jul 2025).
• Two-Tower Model Conversion: For online serving, the sequential transformer is converted into a two-tower ranker. User embeddings are computed offline over historical sequences; candidate items are embedded independently. The scoring function is $\phi_\theta(u, i_t) = \langle h_t, i_t \rangle$ (Khrylchenko et al., 21 Jul 2025).
• Offline-Inference Integration: The output of the two-tower network can be used as a re-ranking feature in larger production systems, with user embeddings refreshed daily, aligning with real-world latency budgets (Khrylchenko et al., 21 Jul 2025).

4. Scaling Laws and Empirical Performance

Scaling behavior in transformer recommenders follows established principles from LLMing:

• Scaling Laws: As model size and training data are increased, metrics such as next-item prediction normalized entropy, feedback prediction entropy, and pairwise accuracy monotically improve. Gains persist up to at least the 1-billion parameter scale (Khrylchenko et al., 21 Jul 2025).
• Empirical Results: In industrial deployment on a large-scale music platform, the one-billion parameter transformer recommender achieves an uplift of +2.26% in total listening time and +6.37% in likelihood of user likes, substantially surpassing baselines and prior deep learning recommender systems (Khrylchenko et al., 21 Jul 2025).
• Context Length: The system demonstrates that longer sequence lengths (up to 8192 core user actions in context) benefit from the attention mechanism, improving predictive power and personalization (Khrylchenko et al., 21 Jul 2025).
• Ablation and Stability: Simplification strategies such as merging input triplets do not lead to performance degradation, and the two-stage pipeline (pretraining and fine-tuning) achieves robust improvements over both smaller and classical architectures (Khrylchenko et al., 21 Jul 2025).

5. Innovations in Large-Scale Sequential Modeling

Recent advances enabling these results include:

• Task Decomposition: Autoregressive learning is decomposed into feedback prediction (learning user taste) and next-item prediction (imitation learning of the logging policy), similar to approaches in offline reinforcement learning (Khrylchenko et al., 21 Jul 2025).
• Input and Output Engineering: By merging context, item, and feedback tokens with tailored projections, the model manages computational cost without losing informativeness. This strategy is essential to reach the 1B-parameter scale efficiently (Khrylchenko et al., 21 Jul 2025).
• Fine-Tuning for Real-World Use: The two-tower adaptation allows daily batch-inference over all users, making the approach compatible with industry operational constraints and inference budgets (Khrylchenko et al., 21 Jul 2025).
• Scaling Beyond Previous Work: The demonstrated framework exceeds prior published transformer recommenders (e.g., HSTU limited to ~176M parameters) by nearly one order of magnitude, providing empirical evidence for scalable, compute-efficient deep recommendation systems (Khrylchenko et al., 21 Jul 2025).

6. Implications and Future Directions

• Personalization at Scale: The ability to pretrain on vast user histories and to scale both parameter count and context length opens up new frontiers for learning nuanced, long-range behavioral patterns.
• Unified Modeling: The framework provides a foundation for integrating multi-modal and contextual features by leveraging the same sequential encoding paradigm.
• Industrial Impact: The architecture has been shown to drive significant uplifts in core product metrics in live, production-scale deployments, demonstrating the practical viability of large transformer recommenders (Khrylchenko et al., 21 Jul 2025).

A plausible implication is that future work may extend these architectures to even larger scales, integrate richer action and multimodal context, and explore joint generative–discriminative objectives for even greater recommendation quality and robustness.

---

Parameter	Approach	Impact
Model size	3.2M – 1B+ params	Consistent quality improvements
Input rep./compression	Triplet merging and MLP	Efficient large-scale input handling
Deployment	Two-tower fine-tuned encoder	Fast, scalable offline/online usage
Training objectives	NIP + FP dual loss	Improved generalization

The scaling of transformer recommenders to the billion-parameter regime establishes their position as the emerging standard for personalization in industrial recommendation systems, provided sufficient infrastructure and data are available (Khrylchenko et al., 21 Jul 2025).