Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs (2407.15431v1)

Published 22 Jul 2024 in cs.SI, cs.AI, and cs.LG

Abstract: The text-attributed graph (TAG) is one kind of important real-world graph-structured data with each node associated with raw texts. For TAGs, traditional few-shot node classification methods directly conduct training on the pre-processed node features and do not consider the raw texts. The performance is highly dependent on the choice of the feature pre-processing method. In this paper, we propose P2TAG, a framework designed for few-shot node classification on TAGs with graph pre-training and prompting. P2TAG first pre-trains the LLM (LM) and graph neural network (GNN) on TAGs with self-supervised loss. To fully utilize the ability of LLMs, we adapt the masked LLMing objective for our framework. The pre-trained model is then used for the few-shot node classification with a mixed prompt method, which simultaneously considers both text and graph information. We conduct experiments on six real-world TAGs, including paper citation networks and product co-purchasing networks. Experimental results demonstrate that our proposed framework outperforms existing graph few-shot learning methods on these datasets with +18.98% ~ +35.98% improvements.

PDF Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. The extreme classification repository: Multi-label datasets and code. http://manikvarma.org/downloads/XC/XMLRepository.html
  2. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In KDD’19. 257–266.
  3. Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. In ICLR’22.
  4. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR’20.
  5. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS’16. 3837–3845.
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT’19 (1). 4171–4186.
  7. Graph prototypical networks for few-shot learning on attributed networks. In CIKM’20.
  8. GLM: General language model pretraining with autoregressive blank infilling. In ACL’22. 320–335.
  9. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML’17.
  10. Inductive Representation Learning on Large Graphs. In NeurIPS’17.
  11. Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In ICML’20. 4116–4126.
  12. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In ICLR’21.
  13. Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph representation learning. arXiv preprint arXiv:2305 (2023).
  14. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW’23. 737–746.
  15. GraphMAE: Self-Supervised Masked Graph Autoencoders. In KDD’22. 594–604.
  16. Kexin Huang and Marinka Zitnik. 2020. Graph meta learning via local subgraphs. NeurIPS’20 (2020).
  17. Prompt-based node feature extractor for few-shot learning on text-attributed graphs. arXiv preprint arXiv:2309.02848 (2023).
  18. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems 33, 2 (2021), 494–514.
  19. Albert: A lite bert for self-supervised learning of language representations. In ICLR’20.
  20. Jure Leskovec and Christos Faloutsos. 2006. Sampling from large graphs. In KDD’06.
  21. Adsgnn: Behavior-graph augmented relevance modeling in sponsored search. In SIGIR’21. 223–232.
  22. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  23. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
  24. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
  25. GCC: Graph contrastive coding for graph neural network pre-training. In KDD’20. 1150–1160.
  26. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198 7 (2020), 15.
  27. Prototypical networks for few-shot learning. In NeurIPS’17.
  28. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD’22. 1717–1727.
  29. All in One: Multi-Task prompting for graph neural networks. In KDD’23.
  30. Large-scale representation learning on graphs via bootstrapping. In ICLR’22.
  31. Graph Attention Networks. In ICLR’18.
  32. Microsoft academic graph: When experts are not enough. Quantitative Science Studies 1, 1 (2020), 396–413.
  33. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022).
  34. Task-adaptive few-shot node classification. In KDD’22.
  35. Max Welling and Thomas N Kipf. 2017. Semi-supervised classification with graph convolutional networks. In ICLR’17.
  36. Zhihao Wen and Yuan Fang. 2023. Augmenting low-Resource text classification with graph-grounded pre-training and prompting. In SIGIR’23.
  37. Simplifying graph convolutional networks. In ICML’19.
  38. Graph wavelet neural network. In ICLR’19.
  39. A comprehensive study on text-attributed graphs: Benchmarking and rethinking. In NeurIPS’23.
  40. Does Graph Distillation See Like Vision Dataset Counterpart?. In NeurIPS’23.
  41. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. In NeurIPS’21.
  42. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS’19. 5754–5764.
  43. Graph few-shot learning via knowledge transfer. In AAAI’20.
  44. LinkBERT: Pretraining Language Models with Document Links. In ACL’22.
  45. Graph convolutional neural networks for web-scale recommender systems. In KDD’18.
  46. Graph Contrastive Learning with Augmentations. In NeurIPS’20.
  47. Empower text-attributed graphs learning with large language models (llms). arXiv preprint arXiv:2310.09872 (2023).
  48. GraphSAINT: Graph Sampling Based Inductive Learning Method. In ICLR’20.
  49. Jiaqi Zeng and Pengtao Xie. 2021. Contrastive self-supervised learning for graph classification. In AAAI’21, Vol. 35. 10824–10832.
  50. From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. In NeurIPS’21. 76–89.
  51. Learning on Large-scale Text-attributed Graphs via Variational Inference. In ICLR’23.
  52. Hierarchical label with imbalance and attributed network structure fusion for network embedding. AI Open 3 (2022), 91–100.
  53. Hierarchical representation learning for attributed networks. IEEE Transactions on Knowledge and Data Engineering 35, 3 (2023), 2641–2656.
  54. Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. In NeurIPS’22.
  55. Meta-gnn: On few-shot node classification in graph meta-learning. In CIKM’2019.
  56. Textgnn: Improving text encoder via graph neural network in sponsored search. In WWW’21. 2848–2857.
  57. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Huanjing Zhao (2 papers)
  2. Beining Yang (6 papers)
  3. Yukuo Cen (19 papers)
  4. Junyu Ren (2 papers)
  5. Chenhui Zhang (16 papers)
  6. Yuxiao Dong (119 papers)
  7. Evgeny Kharlamov (34 papers)
  8. Shu Zhao (31 papers)
  9. Jie Tang (302 papers)
Citations (4)
View on Semantic Scholar