- The paper demonstrates that soft tokens, learned via MLPs from diverse signals, significantly reduce prompt length while preserving semantic richness.
- It employs modular token makers and compression techniques like MLP-based compression and attention pooling to boost training throughput by approximately 200%.
- Empirical results show improved CTR prediction, recall, and engagement metrics, paving the way for scalable, production-ready recommendation systems.
Summary of "Token Factory: Efficiently Integrating Diverse Signals into Large Recommendation Models"
Motivation and Problem Statement
Transformer-based Large Recommendation Models (LRMs) have demonstrated substantial promise in industrial-scale recommender systems. However, integrating heterogeneous traditional signals—such as dense/sparse features and embedding-based representations—into Transformers poses significant efficiency and scalability challenges. Existing methodologies that convert these features into textual tokens ("textualization") result in excessively long prompts, high memory consumption, and elevated computational costs. These factors directly limit the feasible prompt length, truncating user histories and auxiliary signals, and thus restricting modeling capacity.
Token Factory Framework
The Token Factory framework addresses these integration and efficiency challenges by introducing "soft tokens": learned embedding representations generated from traditional signals, which serve as a new modality within LRMs. Unlike "hard" tokens that map discrete vocabulary indices via standard tokenizers, soft tokens are instantiated through learnable transformations (e.g., MLPs) directly from the input features. This process preserves the semantic richness of input signals while managing prompt length and computational complexity.
Architectural Components
- Token Makers: Modular transformation units that ingest various feature types and output a determinate number of soft tokens. Three classes are paramount:
- Watch History (WH) Token Maker: Encodes user interaction sequences.
- Query Feature Token Maker: Encodes user-level/static query features.
- Candidate Feature Token Maker: Encodes item-level or candidate features.
- Transformational Pipeline: Input features are normalized, embedded, concatenated, and projected into fixed-size soft tokens. The number of output tokens N from each Token Maker is empirically controlled, striking a balance between information retention and computational overhead.
- Prompt Integration: Soft tokens are interleaved with (or replace) textual tokens in the prompt, drastically reducing the prompt length. Static tokens for user/query context can be pre-computed and reused across candidate scoring to optimize serving latency.
Sequence Compression
Token Factory incorporates advanced compression techniques to further condense long interaction sequences:
- MLP-Based Compression: Projects long token sequences into a lower-dimensional space via MLPs over the sequence axis.
- Attention Pooling: Applies lightweight attention mechanisms to aggregate information over windows of K items.
This enables the feasible modeling of much longer user histories or more diverse signals within a fixed input budget.
Empirical Evaluation
Experimental Setup
Experiments were conducted using the Gemini-based PLUM model in both ranking (CTR prediction) and generative retrieval tasks on production-scale datasets.
Main Control Arms
- Baseline: Textualized representations (Semantic ID + auxiliary features via custom tokenization).
- Token Factory: All traditional signals mapped into soft tokens.
Key Metrics
- Ranking (CTR Prediction)
- Prompt Length: Reduced from 1536 to 480 tokens.
- Training Throughput: Approximately 200% faster training due to substantially shorter inputs.
- AUC: Comparable to baseline after convergence, with observed initial lag due to randomly initialized Token Makers.
- Batch Size Scaling: Token Factory's efficiency allowed for a larger global batch size, yielding superior early epoch performance.
- Generative Retrieval (Semantic ID Prediction)
- Recall@10: +2% improvement over baseline.
- Online Metrics: Unique Impressions increased by +16.8% (overall), and +67.1% for one-day-fresh videos; Satisfied Watchers and Satisfied Watch Time also improved (+0.04%, +0.05% respectively). No degradation observed in user engagement metrics.
Ablation and Scaling Studies
- Ablations: Demonstrated that when feature richness is constant, soft token and textual SID models achieve similar AUC. Soft tokens outperform in settings with tight prompt budgets, as they allow more history to be represented.
- Compression for Longer History: Extending watch history from 200 to 500 with compression yielded a +0.08% AUC gain, confirming the utility of Token Factory for leveraging more extensive behavioral signals under computational constraints.
Attention Analysis
Attention visualizations revealed that soft token representations enable denser and more uniformly distributed attention across the input sequence, in contrast to the high redundancy and sparsity seen in textualized setups. This pattern suggests improved capacity utilization and more effective information propagation within the Transformer.
Implications and Future Directions
Token Factory demonstrates that heterogeneous, high-dimensional signals can be efficiently assimilated into Transformer-based LRMs using soft token representations. This approach significantly reduces modeling costs, supports longer and richer user histories, and allows for practical deployment of larger training batches—all without performance regression. The consistent gains in item diversity and fresh content discovery, as evidenced in online production metrics, indicate that Token Factory can facilitate more dynamic and exploratory recommendation mechanisms.
Theoretically, Token Factory represents a paradigm shift: instead of reducing modeling complexity by discarding features or truncating histories, it enables the direct absorption of dense, sparse, and sequential/semantic signals while preserving computational tractability. Future research could extend this approach with more advanced token makers (e.g., cross-modal transformers, conditional bottlenecks), hierarchical soft token recursion, or meta-learning optimal compression strategies per user/session.
Conclusion
Token Factory provides a unified, practical, and empirically validated solution for integrating diverse input modalities into industrial-scale LRMs. Its core architecture of learnable soft tokenization—paired with aggressive prompt compression—effectively bridges the efficiency-quality tradeoff inherent in recommender modeling. The framework’s demonstrated gains in training efficiency, retrieval diversity, and user engagement highlight its relevance for production deployment. These findings motivate further exploration of soft token-based approaches for scalable, flexible, and information-rich architectures in large recommendation systems.
