Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data (1807.00311v1)

Abstract: User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.

• The paper introduces kernel-based interactions and explicit product layers to decouple feature interactions and improve gradient sensitivity.
• It demonstrates that models like PIN boosted click-through rates by approximately 34.67% compared to traditional methods.
• Experimental validations on industrial datasets confirm the effectiveness of these deep learning techniques for user response prediction.

Product-based Neural Networks for User Response Prediction

This paper focuses on a critical component of personalized information retrieval and filtering scenarios—user response prediction, specifically in the context of click prediction. The authors address the challenges posed by multi-field categorical data, predominantly transformed into sparse representations via one-hot encoding, which complicates the representation and optimization processes. The paper explores feature engineering and established shallow modeling approaches, transitioning to the promising field of deep neural networks (DNNs) which have garnered interest due to their high capacity and end-to-end training capabilities.

Problem Identification

The research identifies two distinct challenges in existing modeling techniques for user response prediction: the coupled gradient issue in latent vector-based models, like Factorization Machines (FMs), and the insensitive gradient issue in DNN-based models. The coupled gradient issue arises from FMs' oversimplified assumption that features interact uniformly, leading to interdependencies where changes in one feature inadvertently affect others. The insensitive gradient issue reflects the inherent difficulty DNNs face in effectively learning relevant feature interactions due to the vast hypothesis space and the gradient's lack of sensitivity to specific target functions.

Proposed Methodologies

To mitigate these challenges, the authors propose a series of advancements under the umbrella of Product-based Neural Networks (PNNs):

1. Kernel Product for Feature Interactions: Addressing the coupled gradient issue, the authors introduce kernel products to decouple feature interactions, allowing for more nuanced learning of field-aware interactions. They propose adaptations such as Kernel FM (KFM) and Network-in-FM (NIFM), which extend the traditional FM by incorporating kernel-based interaction learning, thereby preserving memory and enhancing model expressiveness.
2. Product Layers in Neural Networks: To tackle the insensitive gradient issue in DNNs, the paper introduces PNNs that incorporate explicit product layers. These layers are designed to extract feature interactions before passing them to the DNN classifier, effectively bridging the gap left by DNNs' implicit interaction modeling. Specific implementations include Inner Product-based Neural Network (IPNN), Kernel Product-based Neural Network (KPNN), and Product-network In Network (PIN).

Experimental Validation

The paper rigorously tests the proposed models using data from a variety of industrial datasets. The results consistently show that PNNs outperform several benchmarks, including linear models like Logistic Regression (LR), tree models like Gradient Boosting Decision Tree (GBDT), and existing neural network implementations like FNN, DeepFM, and CCPM. Notably, PIN achieves the best empirical results, bolstering the authors' hypothesis on the effectiveness of product layers in capturing intricate feature interactions.

Key numerical success is demonstrated in the online A/B test conducted in a real-world setting, where PIN leads to a considerable improvement in Click-Through Rate (CTR) by approximately 34.67%. This not only underscores the practical viability of the proposed models but also signifies a substantial impact on the revenue and user experience facets of personalized information systems.

Implications and Future Work

The implications of this research are twofold: theoretical and practical. Theoretically, it extends the understanding of feature interaction learning in complex categorical data environments, challenging the reliance on traditional shallow models and highlighting the synergy between explicit interaction modeling and deep learning capacities. Practically, the incorporation of kernel and product layers provides a framework for enhancing prediction accuracy in personalized recommendations and advertisement systems.

Future investigations could focus on exploring diverse kernel configurations and micro-network architectures to further refine the balance between model complexity and generalization ability. Additionally, enhancing training algorithms for better handling of sparse data may prove beneficial. This research lays a foundation for more robust AI-driven decision-making in environments where data sparsity and interaction complexity pose significant challenges.