AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction

Abstract: Learning feature interactions is crucial for click-through rate (CTR) prediction in recommender systems. In most existing deep learning models, feature interactions are either manually designed or simply enumerated. However, enumerating all feature interactions brings large memory and computation cost. Even worse, useless interactions may introduce noise and complicate the training process. In this work, we propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS). AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. In the \emph{search stage}, instead of searching over a discrete set of candidate feature interactions, we relax the choices to be continuous by introducing the architecture parameters. By implementing a regularized optimizer over the architecture parameters, the model can automatically identify and remove the redundant feature interactions during the training process of the model. In the \emph{re-train stage}, we keep the architecture parameters serving as an attention unit to further boost the performance. Offline experiments on three large-scale datasets (two public benchmarks, one private) demonstrate that AutoFIS can significantly improve various FM based models. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service, where a 10-day online A/B test demonstrated that AutoFIS improved the DeepFM model by 20.3\% and 20.1\% in terms of CTR and CVR respectively.

• The paper introduces AutoFIS, a two-stage algorithm that uses continuous architecture parameters to automatically select key feature interactions and eliminate redundant computations.
• Extensive offline experiments demonstrate significant AUC improvements and reduced model complexity on multiple benchmarks.
• Online deployment on Huawei's App Store yielded over 20% gains in CTR and CVR, confirming the method's practical industrial impact.

Overview of AutoFIS for Feature Interaction Selection in CTR Prediction

The paper "AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction" explores a significant area of interest within recommender systems—namely, optimizing Click-Through Rate (CTR) predictions. The research addresses the complexity and computational overhead associated with enumerating and processing all potential feature interactions in deep learning models, particularly those utilized in CTR tasks. CTR prediction is crucial for determining user engagement with recommended items such as ads or products. This paper posits that selecting pertinent feature interactions automatically can streamline model efficiency and performance.

The authors introduce AutoFIS, a two-stage algorithm devised to automatically select important feature interactions for factorization models with minimal computational cost. This is achieved by integrating the selection process with the training phase of the target model. This approach stands in contrast to traditional methods, which either manually define interactions or exhaustively evaluate all possibilities, thus incurring substantial memory and computation requirements.

Key Findings and Contributions

1. Architecture Parameters: Instead of overloading the model with discrete sets of candidate feature interactions, AutoFIS employs continuous architecture parameters, which help identify and eliminate redundant interactions during training. It leverages GRDA optimizer to achieve sparse solutions, which aids in removing unnecessary interactions that could introduce noise and hinder convergence.
2. Offline Experiments: Extensive offline experiments on datasets—two public benchmarks and one private—demonstrate the algorithm's ability to significantly enhance CTR prediction performance across several models, including FM and DeepFM. Particularly notable improvements in terms of AUC were observed when AutoFIS was applied, with additional efficiencies in computation time due to reduced model complexity.
3. Online Deployment: AutoFIS was successfully implemented on Huawei's App Store recommendation service, showing remarkable improvements in real-world settings. An online A/B test revealed improvements in CTR and CVR by over 20%, which translates directly to business revenue growth.
4. Transferability: Crucially, the paper discusses how the selected interactions from AutoFIS can be transferred to other deep learning models, demonstrating its flexibility and broader applicability.
5. Ablation Studies: The authors conducted multiple ablation studies to validate the effectiveness of AutoFIS's components, including architecture parameterization, batch normalization, and optimization strategies.

Implications for AI and Future Research

AutoFIS introduces a method to streamline factorization models by cutting down on unnecessary computations, a critical step toward optimizing recommender systems in AI. This approach suggests that similar frameworks could be applied to other areas where feature redundancy is an issue, thus improving efficiency in large-scale machine learning models.

The method hastens the search for high-order feature interactions, providing new directions for research in automatic feature selection and neural architecture search. This could inspire further exploration into gradient-based optimization techniques and their application in trimming complexity while maintaining or improving model performance. Additional research might explore how AutoFIS-like methods could be adapted for different model types beyond CTR prediction, potentially benefiting a range of applications from natural language processing to computer vision and beyond.

In summary, AutoFIS offers a pragmatic and effective approach to reducing the computational burden of feature interaction enumeration in CTR prediction tasks. Its practical deployment demonstrates its industrial relevance, presenting a template for similar advancements in AI-driven recommender systems.