Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Preliminary Roadmap for LLMs as Assistants in Exploring, Analyzing, and Visualizing Knowledge Graphs (2404.01425v1)

Published 1 Apr 2024 in cs.HC

Abstract: We present a mixed-methods study to explore how LLMs can assist users in the visual exploration and analysis of knowledge graphs (KGs). We surveyed and interviewed 20 professionals from industry, government laboratories, and academia who regularly work with KGs and LLMs, either collaboratively or concurrently. Our findings show that participants overwhelmingly want an LLM to facilitate data retrieval from KGs through joint query construction, to identify interesting relationships in the KG through multi-turn conversation, and to create on-demand visualizations from the KG that enhance their trust in the LLM's outputs. To interact with an LLM, participants strongly prefer a chat-based 'widget,' built on top of their regular analysis workflows, with the ability to guide the LLM using their interactions with a visualization. When viewing an LLM's outputs, participants similarly prefer a combination of annotated visuals (e.g., subgraphs or tables extracted from the KG) alongside summarizing text. However, participants also expressed concerns about an LLM's ability to maintain semantic intent when translating natural language questions into KG queries, the risk of an LLM 'hallucinating' false data from the KG, and the difficulties of engineering a 'perfect prompt.' From the analysis of our interviews, we contribute a preliminary roadmap for the design of LLM-driven knowledge graph exploration systems and outline future opportunities in this emergent design space.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. B. Abu-Salih. Domain-specific knowledge graphs: A survey. J. Netw. Comput. Appl., 185:103076, 2021. doi: 10 . 1016/j . jnca . 2021 . 103076
  2. Can knowledge graphs reduce hallucinations in llms?: A survey. arXiv preprint arXiv:2311.07914, 2023.
  3. D. Akbaba and M. Meyer. “two heads are better than one”: Pair-interviews for visualization. In 2023 IEEE Visualization and Visual Analytics (VIS), pp. 206–210. IEEE, 2023.
  4. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031, 2022. doi: 10 . 48550/arXiv . 2204 . 06031
  5. Leancontext: Cost-efficient domain-specific question answering using llms. Natural Language Processing Journal, p. 100065, 2024.
  6. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations, 2023.
  7. Articulate 2 : Toward a conversational interface for visual data exploration. In Proc. VIS, 2016.
  8. The semantic web. Sci. Am., 284(5):34–43, 2001.
  9. Knowledge Graphs: New Directions for Knowledge Representation on the Semantic Web (Dagstuhl Seminar 18371). Dagstuhl Reports, 8(9):29–111, 2019. doi: 10 . 4230/DagRep . 8 . 9 . 29
  10. V. Braun and V. Clarke. Using thematic analysis in psychology. Qual. Res. Psychol., 3(2):77–101, 2006. doi: 10 . 1191/1478088706qp063oa
  11. M. Brehmer and T. Munzner. A multi-level typology of abstract visualization tasks. IEEE transactions on visualization and computer graphics, 19(12):2376–2385, 2013.
  12. Language models are few-shot learners. In Adv. Neural Inf., vol. 33, pp. 1877–1901. Curran Associates, Inc., 2020.
  13. Cava: A visual analytics system for exploratory columnar data augmentation using knowledge graphs. IEEE Trans. Vis. Comput. Graph., 27(2):1731–1741, 2021. doi: 10 . 1109/TVCG . 2020 . 3030443
  14. Knowledge graph completion: A review. Ieee Access, 8:192435–192456, 2020.
  15. Beyond generating code: Evaluating gpt on a data visualization course. In 2023 IEEE VIS Workshop on Visualization Education, Literacy, and Activities (EduVis), pp. 16–21, 2023. doi: 10 . 1109/EduVis60792 . 2023 . 00009
  16. Are two heads better than one in ai-assisted decision making? comparing the behavior and performance of groups and individuals in human-ai collaborative recidivism risk assessment. In In Proc. CHI 2023, CHI ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10 . 1145/3544548 . 3581015
  17. P. Cimiano and H. Paulheim. Knowledge graph refinement: A survey of approaches and evaluation methods. Semant. Web, 8(3):489–508, 2017. doi: 10 . 3233/SW-160218
  18. A multi-modal natural language interface to an information visualization environment. International Journal of Speech Technology, 4:297–314, 2001.
  19. Qualitative and mixed methods provide unique contributions to outcomes research. Circulation, 119(10):1442–1452, 2009.
  20. Don’t just tell me, ask me: Ai systems that intelligently frame explanations as questions improve human logical discernment accuracy over causal ai explanations. In In Proc. CHI 2023, CHI ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10 . 1145/3544548 . 3580672
  21. Developing and using a codebook for the analysis of interview data: An example from a professional development research project. Field Methods, 23(2):136–155, 2011. doi: 10 . 1177/1525822X10388468
  22. Analyza: Exploring data with conversation. In In Proc. ACM IUI, pp. 493–504, 2017.
  23. L. Ehrlinger and W. Wöß. Towards a definition of knowledge graphs. Proc. ESWC Posters and Demos Track, 48(1-4):2, 2016.
  24. How large language models will disrupt data management. In Proc. VLDB, 16(11):3302–3309, 2023.
  25. Cypher: An evolving query language for property graphs. In Proc. SIGMOD, p. 1433–1445. ACM, New York, 2018. doi: 10 . 1145/3183713 . 3190657
  26. A. Gal. Uncertain entity resolution: Re-evaluating entity resolution in the big data era: Tutorial. Proc. VLDB Endow., 7(13):1711–1712, 2014. doi: 10 . 14778/2733004 . 2733068
  27. Natural SQL: Making SQL easier to infer from natural language specifications. In Proc. EMNLP, pp. 2030–2042. ACL, Punta Cana, Dominican Republic, 2021. doi: 10 . 18653/v1/2021 . findings-emnlp . 174
  28. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
  29. Dialect prejudice predicts ai decisions about people’s character, employability, and criminality. arXiv preprint arXiv:2403.00742, 2024.
  30. Knowledge graphs. ACM Comput. Surv., 54(4), 2021. doi: 10 . 1145/3447772
  31. Human factors in model interpretability: Industry practices, challenges, and needs. Proc. CHI, 4(CSCW1), 2020. doi: 10 . 1145/3392878
  32. Flownl: Asking the flow data in natural languages. IEEE Trans. Vis. Comput. Graph., 29(1):1200–1210, 2023. doi: 10 . 1109/TVCG . 2022 . 3209453
  33. Large language models versus natural language understanding and generation. In In Proc. PCI, pp. 278–290, 2023.
  34. V. Kepuska and G. Bohouta. Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC), pp. 99–103. IEEE, 2018.
  35. F. C. Kintzer. Advantages of open-response questions in survey research. Community Junior College Research Quarterly, 2(1):37–46, 1977.
  36. Bringing Graph Databases and Network Visualization Together (Dagstuhl Seminar 22031). Dagstuhl Reports, 12(1):67–82, 2022. doi: 10 . 4230/DagRep . 12 . 1 . 67
  37. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
  38. F. Lecue. On the role of knowledge graphs in explainable AI. Semant. Web, 11(1):41–51, 2020. doi: 10 . 3233/SW-190374
  39. F. Li and H. V. Jagadish. Constructing an interactive natural language interface for relational databases. In Proc. VLDB, 8(1):73–84, 2014.
  40. Knowledge graphs in practice: Characterizing their users, challenges, and visualization opportunities. IEEE Transactions on Visualization and Computer Graphics, 30(1):584–594, 2024. doi: 10 . 1109/TVCG . 2023 . 3326904
  41. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. arXiv preprint arXiv:2305.03111, 2023.
  42. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv preprint arXiv:2403.10446, 2024.
  43. Inksight: Leveraging sketch interaction for documenting chart findings in computational notebooks. IEEE Transactions on Visualization and Computer Graphics, 30(1):944–954, 2024. doi: 10 . 1109/TVCG . 2023 . 3327170
  44. Knowledge graph exploration systems: are we lost? In CIDR, vol. 22, pp. 10–13, 2022.
  45. Graph-query suggestions for knowledge graph exploration. In In Proc. ACM WWW, pp. 2549–2555, 2020.
  46. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023.
  47. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061, 2023.
  48. Codebook development for team-based qualitative analysis. CAM j., 10(2):31–36, 1998. doi: 10 . 1177/1525822X980100020301
  49. Knowledge injection to counter large language model (llm) hallucination. In European Semantic Web Conference, pp. 182–185. Springer, 2023.
  50. Knowledge injection to counter large language model (llm) hallucination. In C. Pesquita, H. Skaf-Molli, V. Efthymiou, S. Kirrane, A. Ngonga, D. Collarana, R. Cerqueira, M. Alam, C. Trojahn, and S. Hertling, eds., The Semantic Web: ESWC 2023 Satellite Events, pp. 182–185. Springer Nature Switzerland, Cham, 2023.
  51. Facilitating conversational interaction in natural language interfaces for visualization. In Proc. VIS, pp. 6–10, 2022. doi: 10 . 1109/VIS54862 . 2022 . 00010
  52. Snowball sampling: A purposeful method of sampling in qualitative research. Strides in development of medical education, 14(3), 2017.
  53. Using an llm to help with code understanding. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pp. 881–881. IEEE Computer Society, 2024.
  54. NL4DV: A Toolkit for generating Analytic Specifications for Data Visualization from Natural Language queries. IEEE Trans. Vis. Comput. Graph., 2020. doi: 10 . 1109/TVCG . 2020 . 3030378
  55. Sorry, i don’t speak sparql: translating sparql queries into natural language. In In Proc. ACM WWW, WWW ’13, p. 977–988. Association for Computing Machinery, New York, NY, USA, 2013. doi: 10 . 1145/2488388 . 2488473
  56. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, pp. 1–20, 2024. doi: 10 . 1109/TKDE . 2024 . 3352100
  57. Language models as knowledge bases? In Proc. EMNLP/IJCNLP, pp. 2463–2473. ACL, Hong Kong, 2019. doi: 10 . 18653/v1/D19-1250
  58. P. Pirolli and S. Card. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In In Proc. International Conf on Intelligence Analysis, vol. 5, pp. 2–4. McLean, VA, USA, 2005.
  59. F. Rabiee. Focus-group interview and data analysis. In Proc. NUTR SOC, 63(4):655–660, 2004.
  60. Collaborating with a text-based chatbot: An exploration of real-world collaboration strategies enacted during human-chatbot interactions. In In Proc. CHI 2023, CHI ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10 . 1145/3544548 . 3580995
  61. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023.
  62. Code llama: Open foundation models for code, 2024.
  63. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In Proc. CHI. ACM, New York, 2021. doi: 10 . 1145/3411764 . 3445518
  64. Knowledge graph-augmented language models for complex question answering. In B. Dalvi Mishra, G. Durrett, P. Jansen, D. Neves Ribeiro, and J. Wei, eds., In Proc. NLRSE, pp. 1–8. Association for Computational Linguistics, Toronto, Canada, June 2023. doi: 10 . 18653/v1/2023 . nlrse-1 . 1
  65. Eviza: A natural language interface for visual analysis. In In Proc. ACM UIST, pp. 365–377, 2016.
  66. Towards natural language interfaces for data visualization: A survey. IEEE transactions on visualization and computer graphics, 2022.
  67. Data player: Automatic generation of data videos with narration-animation interplay. IEEE Transactions on Visualization and Computer Graphics, 30(1):109–119, 2024. doi: 10 . 1109/TVCG . 2023 . 3327197
  68. A. Srinivasan and V. Setlur. Snowy: Recommending utterances for conversational visual analysis. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 864–880, 2021.
  69. N. Sultanum and A. Srinivasan. Datatales: Investigating the use of large language models for authoring data-driven articles. In 2023 IEEE Visualization and Visual Analytics (VIS), pp. 231–235. IEEE, 2023.
  70. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697, 2023.
  71. Knowledge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges, vol. 47 of Studies on the Semantic Web. IOS Press, 2020.
  72. Troubles with nulls, views from the users. Proc. VLDB Endow., 15(11):2613–2625, 2022. doi: 10 . 14778/3551793 . 3551818
  73. J. W. Tukey et al. Exploratory data analysis, vol. 2. Reading, MA, 1977.
  74. P.-P. Vázquez. Are llms ready for visualization? arXiv preprint arXiv:2403.06158, 2024.
  75. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds., Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837. Curran Associates, Inc., 2022.
  76. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint arXiv:2308.09729, 2023.
  77. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.
  78. Llm-based sparql generation with selected schema from large scale knowledge base. In China Conference on Knowledge Graph and Semantic Computing, pp. 304–316. Springer, 2023.
  79. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193, 2019.
  80. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226, 2023.
  81. Neural, symbolic and neural-symbolic reasoning on knowledge graphs. AI Open, 2:14–35, 2021.
  82. "it’s a fair game”, or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents. arXiv preprint arXiv:2309.11653, 2023.
  83. Telling stories from computational notebooks: Ai-assisted presentation slides creation for presenting data science work. In In Proc. CHI 2022, pp. 1–20, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Harry Li (8 papers)
  2. Gabriel Appleby (13 papers)
  3. Ashley Suh (18 papers)
Citations (2)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets