Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robust Training of Temporal GNNs using Nearest Neighbours based Hard Negatives (2402.09239v1)

Published 14 Feb 2024 in cs.LG and cs.IR

Abstract: Temporal graph neural networks Tgnn have exhibited state-of-art performance in future-link prediction tasks. Training of these TGNNs is enumerated by uniform random sampling based unsupervised loss. During training, in the context of a positive example, the loss is computed over uninformative negatives, which introduces redundancy and sub-optimal performance. In this paper, we propose modified unsupervised learning of Tgnn, by replacing the uniform negative sampling with importance-based negative sampling. We theoretically motivate and define the dynamically computed distribution for a sampling of negative examples. Finally, using empirical evaluations over three real-world datasets, we show that Tgnn trained using loss based on proposed negative sampling provides consistent superior performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Privacy-Aware Recommender Systems Challenge on Twitter’s Home Timeline. https://doi.org/10.48550/ARXIV.2004.13715
  2. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014).
  3. CuCo: Graph Representation with Curriculum Contrastive Learning. In International Joint Conference on Artificial Intelligence.
  4. Debiased contrastive learning. Advances in Neural Information Processing Systems 33 (2020).
  5. Provably expressive temporal graph networks. ArXiv abs/2209.15059 (2022).
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805
  7. DynGEM: Deep Embedding Method for Dynamic Graphs. ArXiv abs/1805.11273 (2018).
  8. Alex Graves. 2012. Long short-term memory. Supervised sequence labelling with recurrent neural networks (2012), 37–45.
  9. Shubham Gupta and Srikanta Bedathur. 2022. A Survey on Temporal Graph Representation Learning and Generative Modeling. arXiv:2208.12126 [cs.LG]
  10. TIGGER: Scalable Generative Modelling for Temporal Interaction Graphs. https://doi.org/10.48550/ARXIV.2203.03564
  11. GRAFENNE: Learning on Graphs with Heterogeneous and Dynamic Feature Sets. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 12165–12181. https://proceedings.mlr.press/v202/gupta23b.html
  12. Inductive Representation Learning on Large Graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 1025–1035.
  13. Chih-Hui Ho and Nuno Nvasconcelos. 2020. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems 33 (2020), 17081–17093.
  14. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. https://doi.org/10.48550/ARXIV.2104.06967
  15. Yifei Hu and Ya Zhang. 2021. Graph Contrastive Learning with Local and Global Mutual Information Maximization. In Proceedings of the 2020 8th International Conference on Information Technology: IoT and Smart City (Xi’an, China) (ICIT ’20). Association for Computing Machinery, New York, NY, USA, 74–78. https://doi.org/10.1145/3446999.3447013
  16. A survey on contrastive self-supervised learning. Technologies 9, 1 (2020), 2.
  17. Glen Jeh and Jennifer Widom. 2003. Scaling Personalized Web Search. In Proceedings of the 12th International Conference on World Wide Web (Budapest, Hungary) (WWW ’03). Association for Computing Machinery, New York, NY, USA, 271–279. https://doi.org/10.1145/775152.775191
  18. Hard Negative Mixing for Contrastive Learning. ArXiv abs/2010.01028 (2020).
  19. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550
  20. Angelos Katharopoulos and François Fleuret. 2018. Not All Samples Are Created Equal: Deep Learning with Importance Sampling. In ICML.
  21. Time2Vec: Learning a Vector Representation of Time. https://doi.org/10.48550/ARXIV.1907.05321
  22. Community interaction and conflict on the web. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 933–943.
  23. Edge weight prediction in weighted signed networks. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, 221–230.
  24. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 1269–1278.
  25. In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). Association for Computational Linguistics, Online, 163–173. https://doi.org/10.18653/v1/2021.repl4nlp-1.17
  26. Towards Fine-Grained Temporal Network Representation via Time-Reinforced Random Walk. In AAAI.
  27. Temporal Network Embedding with Micro- and Macro-Dynamics. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19). Association for Computing Machinery, New York, NY, USA, 469–478. https://doi.org/10.1145/3357384.3357943
  28. Image-Based Recommendations on Styles and Substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (Santiago, Chile) (SIGIR ’15). Association for Computing Machinery, New York, NY, USA, 43–52. https://doi.org/10.1145/2766462.2767755
  29. EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.
  30. Graph Representation Learning via Graphical Mutual Information Maximization. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 259–270. https://doi.org/10.1145/3366423.3380112
  31. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 5835–5847. https://doi.org/10.18653/v1/2021.naacl-main.466
  32. RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2825–2835. https://doi.org/10.18653/v1/2021.emnlp-main.224
  33. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 (2020).
  34. Temporal Graph Networks for Deep Learning on Dynamic Graphs. In ICML 2020 Workshop on Graph Representation Learning.
  35. Aravind Sankar and Hao Yang. 2020. DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks. In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM ’20). Association for Computing Machinery, New York, NY, USA, 519–527. https://doi.org/10.1145/3336191.3371845
  36. Node Embedding over Temporal Graphs. In IJCAI.
  37. DyRep: Learning Representations over Dynamic Graphs. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=HyePrhR5KX
  38. Laurens van der Maaten and Geoffrey E. Hinton. 2008. Visualizing Data using t-SNE. In Journal of Machine Learning Research, Vol. 9. 2579–2605.
  39. Graph Attention Networks. International Conference on Learning Representations (2018). https://openreview.net/forum?id=rJXMpikCZ
  40. Deep Graph Infomax. In International Conference on Learning Representations. https://openreview.net/forum?id=rklz9iAcKQ
  41. Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks. CoRR abs/2101.05974 (2021). arXiv:2101.05974 https://arxiv.org/abs/2101.05974
  42. ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning. In International conference on machine learning. PMLR.
  43. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=zeFrfgyZln
  44. Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07962 (2020).
  45. Self-supervised Graph-level Representation Learning with Local and Global Structure. In International Conference on Machine Learning.
  46. Understanding Negative Sampling in Graph Representation Learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 1666–1676. https://doi.org/10.1145/3394486.3403218
  47. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 974–983. https://doi.org/10.1145/3219819.3219890
  48. Optimizing Dense Retrieval Model Training with Hard Negatives. https://doi.org/10.48550/ARXIV.2104.08051
  49. Learning Node Embeddings in Interaction Graphs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM ’17). Association for Computing Machinery, New York, NY, USA, 397–406. https://doi.org/10.1145/3132847.3132918
  50. Zhen Zhang and Can Wang. 2020. Learning Temporal Interaction Graph Embedding via Coupled Memory Networks. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 3049–3055. https://doi.org/10.1145/3366423.3380076
  51. Graph Debiased Contrastive Learning with Joint Representation Clustering. In International Joint Conference on Artificial Intelligence.
  52. Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning. ArXiv abs/2108.13886 (2021).
  53. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond. http://arxiv.org/abs/2006.04131
  54. Link and Triadic Closure Delay: Temporal Metrics for Social Network Dynamics. Proceedings of the International AAAI Conference on Web and Social Media 8, 1 (May 2014), 564–573. https://ojs.aaai.org/index.php/ICWSM/article/view/14507
  55. Yuan Zuo and Junjie Wu. 2018. Embedding Temporal Network via Neighborhood Formation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 2857–2866. https://doi.org/10.1145/3219819.3220054
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets