Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation (2408.00801v1)

Abstract: Factorization machine (FM) variants are widely used in recommendation systems that operate under strict throughput and latency requirements, such as online advertising systems. FMs are known both due to their ability to model pairwise feature interactions while being resilient to data sparsity, and their computational graphs that facilitate fast inference and training. Moreover, when items are ranked as a part of a query for each incoming user, these graphs facilitate computing the portion stemming from the user and context fields only once per query. Consequently, in terms of inference cost, the number of user or context fields is practically unlimited. More advanced FM variants, such as FwFM, provide better accuracy by learning a representation of field-wise interactions, but require computing all pairwise interaction terms explicitly. The computational cost during inference is proportional to the square of the number of fields, including user, context, and item. When the number of fields is large, this is prohibitive in systems with strict latency constraints. To mitigate this caveat, heuristic pruning of low intensity field interactions is commonly used to accelerate inference. In this work we propose an alternative to the pruning heuristic in FwFMs using a diagonal plus symmetric low-rank decomposition. Our technique reduces the computational cost of inference, by allowing it to be proportional to the number of item fields only. Using a set of experiments on real-world datasets, we show that aggressive rank reduction outperforms similarly aggressive pruning, both in terms of accuracy and item recommendation speed. We corroborate our claim of faster inference experimentally, both via a synthetic test, and by having deployed our solution to a major online advertising system. The code to reproduce our experimental results is at https://github.com/michaelviderman/pytorch-fm/tree/dev.

• The paper introduces a DPLR decomposition that transforms quadratic FwFM inference into linear complexity.
• Experiments on datasets like Criteo and MovieLens show comparable accuracy with significantly reduced latency.
• The DPLR method offers a practical solution for real-time recommendation systems, enhancing ad targeting and user experience.

Insightful Overview of "Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation"

The paper "Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation" by Shtoff et al. addresses a computational challenge in recommendation systems, particularly those that operate under strict latency constraints such as online advertising systems. Factorization Machines (FMs) and their advanced variants like field-weighted FMs (FwFMs) are known for their ability to model pairwise feature interactions effectively. However, FwFMs typically suffer from high computational costs during inference due to their quadratic complexity concerning the number of fields.

Contributions and Methods

The main contribution of this paper is the introduction of the Diagonal Plus Low-Rank (DPLR) decomposition for FwFMs, coined as DPLR-FwFM. The authors propose a method to reduce the computational costs associated with FwFMs while maintaining or even improving model accuracy. The idea is to decompose the field interaction matrix into a sum of a diagonal matrix and a low-rank matrix. This decomposition enables the reformulation of the pairwise feature interaction terms, allowing efficient computation while retaining the benefits of FwFMs in terms of prediction accuracy.

Key Findings:

1. Theoretical Foundations: The authors show mathematically how the DPLR reformulation can transform the original FwFM computational complexity from quadratic to linear concerning the number of item fields, scaled by a small constant factor related to the rank of the low-rank matrix.
2. Experimental Validation: Using real-world datasets (Criteo, Avazu, and MovieLens), the DPLR-FwFM models exhibit improved or comparable predictive performance while significantly reducing inference latency compared to the pruned models.
3. Latency Improvements: The synthetic timing benchmarks and experiments in an actual online advertising environment demonstrate that DPLR-FwFM models provide substantial latency improvements. For instance, when ranking items, the DPLR models outperformed the pruned models by approximately 20-30% in terms of inference time and up to 5% in overall query latency.
4. Post-Hoc Factorization: The paper also explores a "post-hoc" approach, where a low-rank approximation is computed after training a full FwFM model. This approach is less effective, indicating that the DPLR-FwFM should be trained directly to achieve optimal performance.

Implications

The practical implications of this research are significant for industries relying on low-latency recommendation systems, such as real-time advertising. The proposed DPLR-FwFM model allows for faster and more efficient item recommendations without sacrificing accuracy. This can lead to improvements in user experience through quicker response times and potentially better-targeted recommendations, ultimately driving higher engagement and revenue for service providers.

Future Directions

The authors suggest potential avenues for future development:

• Field Importance Quantification: The DPLR decomposition introduces a notion of "field importance," whereby fields with negligible influence can be identified and potentially discarded to further improve efficiency.
• Broader Applications: While the current paper focuses on FwFMs, the principles of DPLR decomposition could be applied to other advanced FM variants or even broader machine learning models that require efficient handling of high-dimensional interactions.

Conclusion

The paper "Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation" presents a well-substantiated advancement in machine learning for recommendation systems. By employing a DPLR decomposition, the authors successfully mitigate the latency issues inherent in FwFMs, achieving a balance between computational efficiency and model performance. This innovation is poised to influence not only the field of recommendation systems but also other domains requiring efficient, scalable interaction modeling. The provided open-source code ensures reproducibility and encourages further exploration and implementation of these techniques in real-world settings.