LiGNN: Graph Neural Networks at LinkedIn (2402.11139v1)

Abstract: In this paper, we present LiGNN, a deployed large-scale Graph Neural Networks (GNNs) Framework. We share our insight on developing and deployment of GNNs at large scale at LinkedIn. We present a set of algorithmic improvements to the quality of GNN representation learning including temporal graph architectures with long term losses, effective cold start solutions via graph densification, ID embeddings and multi-hop neighbor sampling. We explain how we built and sped up by 7x our large-scale training on LinkedIn graphs with adaptive sampling of neighbors, grouping and slicing of training data batches, specialized shared-memory queue and local gradient optimization. We summarize our deployment lessons and learnings gathered from A/B test experiments. The techniques presented in this work have contributed to an approximate relative improvements of 1% of Job application hearing back rate, 2% Ads CTR lift, 0.5% of Feed engaged daily active users, 0.2% session lift and 0.1% weekly active user lift from people recommendation. We believe that this work can provide practical solutions and insights for engineers who are interested in applying Graph neural networks at large scale.

• The paper introduces LiGNN, a scalable graph neural network framework that cuts training time by 7x through adaptive sampling and Kubernetes integration.
• It leverages graph densification, multi-hop sampling, and a novel temporal modeling approach to effectively learn from dynamic interactions.
• The deployment significantly enhances LinkedIn’s recommendation systems, boosting job applications, ad clicks, and overall user engagement.

Deploying Large-Scale Graph Neural Networks at LinkedIn: Insights and Innovations

Introduction

LinkedIn, the world's largest professional network, has successfully integrated Graph Neural Networks (GNNs) at an unprecedented scale with its LiGNN framework. This integration aims to optimize LinkedIn's ecosystem, a complex web of professional connections and interactions among members, companies, and various entities. LiGNN encompasses sophisticated algorithmic advancements and strategic deployment mechanisms that address the unique challenges of applying GNNs in a dynamic, large-scale environment.

Challenges and Technological Advancements

LiGNN's development and deployment at LinkedIn presented numerous challenges, notably GNN training scalability, the integration of diverse entities into a unified graph embedding space, addressing the "cold start" problem, and adapting to the platform's dynamic nature. Through inventive approaches, including adaptive sampling and temporal graph architectures, LiGNN has achieved remarkable efficiency and marked improvements in various recommendation systems on LinkedIn.

GNN Training at Scale

LiGNN's training infrastructure utilizes a Kubernetes-based cluster, integrating Microsoft's DeepGNN graph engine for efficient real-time graph sampling. Implementations such as adaptive neighbor sampling and specialized data processing mechanisms have significantly improved GNN training stability and reduced training times by a factor of 7.

Graph Densification and Multi-Hop Sampling

To mitigate issues of graph sparsity and enhance representation learning, LiGNN employs graph densification techniques and multi-hop sampling strategies. These methods have proven effective in enriching graph representations and accelerating the sampling process, demonstrating notable improvements in various production applications.

Temporal Modeling

The framework introduces a simplified and scalable approach to temporal graph modeling, distinct from conventional temporal GNNs. By modifying neighbor sampling to encapsulate temporal dynamics, and incorporating transformer models for sequence encoding, LiGNN captures time-sensitive interactions, contributing to improved model performance across multiple use cases.

Deployment Lessons and Near-Line Inference Architecture

The deployment of LiGNN has provided critical insights, particularly in addressing feature freshness through a GNN model inference pipeline for near real-time embedding generation. Challenges related to integrating LiGNN with LinkedIn's systems prompted the development of specific Apache Beam components, enhancing the pipeline's efficiency and applicability.

Impact Across LinkedIn

LiGNN's deployment across various domains within LinkedIn, including job and people recommendations, ads, and feed post recommendations, has shown substantial improvements. For instance, LiGNN contributed to increases in job application hearing back rates, ad click-through rates, and overall user engagement metrics. The framework demonstrates the practical value and wide applicability of GNNs in enhancing recommendation systems at scale.

Conclusion

The development and deployment of LiGNN at LinkedIn highlight the potential of graph neural networks in transforming large-scale recommendation systems. By addressing key challenges through innovative approaches to training scalability, cold start solutions, and temporal modeling, LiGNN has achieved significant improvements in various application domains. This work serves as a valuable reference for engineers and researchers interested in applying GNNs at scale, promising further advancements and broader applications in the field of AI.