Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Distilling Event Sequence Knowledge From Large Language Models (2401.07237v3)

Published 14 Jan 2024 in cs.CL and cs.AI

Abstract: Event sequence models have been found to be highly effective in the analysis and prediction of events. Building such models requires availability of abundant high-quality event sequence data. In certain applications, however, clean structured event sequences are not available, and automated sequence extraction results in data that is too noisy and incomplete. In this work, we explore the use of LLMs to generate event sequences that can effectively be used for probabilistic event model construction. This can be viewed as a mechanism of distilling event sequence knowledge from LLMs. Our approach relies on a Knowledge Graph (KG) of event concepts with partial causal relations to guide the generative LLM for causal event sequence generation. We show that our approach can generate high-quality event sequences, filling a knowledge gap in the input KG. Furthermore, we explore how the generated sequences can be leveraged to discover useful and more complex structured knowledge from pattern mining and probabilistic event models. We release our sequence generation code and evaluation framework, as well as corpus of event sequence data.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Topic Detection and Tracking Pilot Study Final Report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, February, 1998., 1 1998.
  2. James Allan. Topic Detection and Tracking: Event-Based Information Organization. Springer Publishing Company, Incorporated, 2012.
  3. On prediction using variable order markov models. J. Artif. Intell. Res., 22:385–421, 2004.
  4. Summary markov models for event sequences. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4836–4842. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
  5. Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006.
  6. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  7. Event prediction from news text using subgraph embedding and graph sequence mining. World Wide Web, 25(6):2403–2428, 2022.
  8. Event-centric natural language processing. In ACL, 2021.
  9. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada, July 2023. Association for Computational Linguistics.
  10. Scaling instruction-finetuned language models, 2022.
  11. RESIN-11: Schema-guided event prediction for 11 newsworthy scenarios. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, pages 54–63, Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics.
  12. Automatic creation of domain templates. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 207–214, Sydney, Australia, July 2006. Association for Computational Linguistics.
  13. A survey of sequential pattern mining. Data Science and Pattern Recognition, 1(1):54–77, 2017.
  14. Examining the state-of-the-art in news timeline summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1322–1334, Online, July 2020. Association for Computational Linguistics.
  15. Knowledge-based news event analysis and forecasting toolkit. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5904–5907. ijcai.org, 2022.
  16. Annollm: Making large language models to be better crowdsourced annotators, 2023.
  17. Long short-term memory. Neural Comput., 9(8):1735–1780, nov 1997.
  18. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. In AAAI, 2021.
  19. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada, July 2023. Association for Computational Linguistics.
  20. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5591–5606, Toronto, Canada, July 2023. Association for Computational Linguistics.
  21. A taxonomy of sequential pattern mining algorithms. ACM Comput. Surv., 43(1), dec 2010.
  22. Discovery of frequent episodes in event sequences. Data Mining and Knowledge Discovery, 1:259–289, 1997.
  23. Sequential pattern mining – approaches and algorithms. ACM Comput. Surv., 45(2), mar 2013.
  24. A survey on event-based news narrative extraction. ACM Comput. Surv., 55(14s), jul 2023.
  25. Learning causality for news events prediction. In Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012, pages 909–918. ACM, 2012.
  26. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020.
  27. A. Raftery. A model for high-order Markov chains. Journal of the Royal Statistical Society, Series B, 47(3):528–539, 1985.
  28. A survey on narrative extraction from textual data. Artif. Intell. Rev., 56(8):8393–8435, 2023.
  29. Mining sequential patterns: Generalizations and performance improvements. In Peter Apers, Mokrane Bouzeghoub, and Georges Gardarin, editors, Advances in Database Technology — EDBT ’96, pages 1–17, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg.
  30. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  31. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78–85, sep 2014.
  32. Revisiting relation extraction in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15566–15589, Toronto, Canada, July 2023. Association for Computational Linguistics.
  33. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
  34. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics.
  35. A survey of event extraction from text. IEEE Access, 7:173111–173137, 2019.
  36. A survey of knowledge-enhanced text generation. ACM Comput. Surv., 54(11s), nov 2022.
  37. Mohammed J. Zaki. Spade: An efficient algorithm for mining frequent sequences. Machine Learning, 42:31–60, 2004.
  38. LMTurk: Few-shot learners as crowdsourcing workers in a language-model-as-a-service framework. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 675–692, Seattle, United States, July 2022. Association for Computational Linguistics.
  39. Liang Zhao. Event prediction in the big data era: A systematic survey. ACM Comput. Surv., 54(5), may 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Somin Wadhwa (9 papers)
  2. Oktie Hassanzadeh (16 papers)
  3. Debarun Bhattacharjya (17 papers)
  4. Ken Barker (8 papers)
  5. Jian Ni (22 papers)