Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Breathing New Life into Existing Visualizations: A Natural Language-Driven Manipulation Framework (2404.06039v1)

Published 9 Apr 2024 in cs.HC

Abstract: We propose an approach to manipulate existing interactive visualizations to answer users' natural language queries. We analyze the natural language tasks and propose a design space of a hierarchical task structure, which allows for a systematic decomposition of complex queries. We introduce a four-level visualization manipulation space to facilitate in-situ manipulations for visualizations, enabling a fine-grained control over the visualization elements. Our methods comprise two essential components: the natural language-to-task translator and the visualization manipulation parser. The natural language-to-task translator employs advanced NLP techniques to extract structured, hierarchical tasks from natural language queries, even those with varying degrees of ambiguity. The visualization manipulation parser leverages the hierarchical task structure to streamline these tasks into a sequence of atomic visualization manipulations. To illustrate the effectiveness of our approach, we provide real-world examples and experimental results. The evaluation highlights the precision of our natural language parsing capabilities and underscores the smooth transformation of visualization manipulations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Low-level components of analytic activity in information visualization. In Proceedings of IEEE Symposium on Information Visualization. 111–117.
  2. Understanding Data Videos: Looking at Narrative Visualization through the Cinematography Lens. In Proceedings of the Annual ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1459–1468.
  3. Articulate2: Toward a conversational interface for visual data exploration. In Proceedings of IEEE Visualization Poster.
  4. Beagle: Automated extraction and interpretation of visualizations from the web. In Proceedings of ACM Conference on Human Factors in Computing Systems. 1–8.
  5. D33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT: Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2301–2309.
  6. Matthew Brehmer and Tamara Munzner. 2013. A Multi-Level Typology of Abstract Visualization Tasks. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2376–2385.
  7. Language Models are Few-Shot Learners. CoRR abs/2005.14165 (2020). arXiv:2005.14165
  8. A Multi-Modal Natural Language Interface to an Information Visualization Environment. International Journal of Speech Technology 4, 3-4 (2001), 297–314.
  9. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 4171–4186.
  10. Quda: Natural Language Queries for Visual Data Analytics. CoRR abs/2005.03257 (2020). arXiv:2005.03257
  11. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proceedings of the Annual ACM Symposium on User Interface Software & Technology. 489–500.
  12. Jonathan Harper and Maneesh Agrawala. 2014. Deconstructing and restyling D3 visualizations. In Proceedings of ACM Symposium on User Interface Software and Technology. 253–262.
  13. Jonathan Harper and Maneesh Agrawala. 2018. Converting Basic D3 Charts into Reusable Style Templates. IEEE Transactions on Visualization and Computer Graphics 24, 3 (2018), 1274–1286.
  14. Jeffrey Heer and George Robertson. 2007. Animated transitions in statistical data graphics. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1240–1247.
  15. Applying pragmatics principles for interaction with visual analytics. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 309–318.
  16. FigureQA: An Annotated Figure Dataset for Visual Reasoning. arXiv preprint arXiv:1710.07300 (2017).
  17. Answering Questions about Charts and Generating Visual Explanations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–13.
  18. Automatic Annotation Synchronizing with Textual Description for Visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA, 1–13.
  19. AutoTitle: An Interactive Title Generator for Visualizations. IEEE Transactions on Visualization and Computer Graphics, early access (2023), 1–12.
  20. ADVISor: Automatic Visualization Answer for Natural-Language Question on Tabular Data. In Proceedings of IEEE Pacific Visualization Symposium. 6–15.
  21. AutoCaption: An Approach to Generate Natural Language Description from Visualization Automatically. In Proceedings of the IEEE Pacific Visualization Symposium. 191–195.
  22. A Spatial Constraint Model for Manipulating Static Visualizations. ACM Transactions on Interative Intelligence Systems (2023), 1 – 30.
  23. Interaction+: Interaction enhancement for web-based visualizations. In Proceedings of IEEE Pacific Visualization Symposium. 61–70.
  24. DeepEye: Towards Automatic Data Visualization. In Proceedings of the IEEE International Conference on Data Engineering. 101–112.
  25. Synthesizing Natural Language to Visualization (NL2VIS) Benchmarks from NL2SQL Benchmarks. In Proceedings of the International Conference on Management of Data. 1235–1247.
  26. Natural language to visualization by neural machine translation. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 217–226.
  27. Tamara Munzner. 2014. Visualization Analysis and Design.
  28. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2021), 369–379.
  29. OpenAI. 2023. ChatGPT API. https://beta.openai.com/docs/api-reference/introduction. Accessed: April 1, 2023.
  30. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 140 (2020), 1–67.
  31. Vega-Lite: A Grammar of Interactive Graphics. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 341–350.
  32. Kamran Sedig and Paul Parsons. 2013. Interaction design for complex cognitive activities with visual representations: A pattern-based approach. AIS Transactions on Human-Computer Interaction 5, 2 (2013), 84–133.
  33. Eviza: A Natural Language Interface for Visual Analysis. In Proceedings of ACM Symposium on User Interface Software and Technology. 365–377.
  34. Towards Natural Language Interfaces for Data Visualization: A Survey. IEEE Transactions on Visualization and Computer Graphics 29, 6 (2023), 3121–3144.
  35. Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay. IEEE Transactions on Visualization and Computer Graphics 30, 1 (2024), 109–119.
  36. Talk2Data: High-Level Question Decomposition for Data-Oriented Question and Answering. CoRR abs/2107.14420 (2021).
  37. InChorus: Designing consistent multimodal interactions for data visualization on tablet devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–13.
  38. Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA, Article 464, 10 pages.
  39. Arjun Srinivasan and John Stasko. 2017. Natural Language Interfaces for Data Analysis with Visualization: Considering What Has and Could Be Asked. In Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers. Goslar, DEU, 55–59.
  40. Articulate: A Semi-Automated Model for Translating Natural Language Queries into Meaningful Visualizations. In Proceedings of International Symposium on Smart Graphics. 184–195.
  41. Towards Natural Language-Based Visualization Authoring. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2023), 1222–1232.
  42. Transformers: State-of-the-art natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 38–45.
  43. SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. CoRR abs/1711.04436 (2017). arXiv:1711.04436
  44. Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1224–1231.
  45. Bowen Yu and Cláudio T. Silva. 2020. FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2020), 1–11.
  46. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 3911–3921.
  47. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR abs/1709.00103 (2017).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Can Liu (40 papers)
  2. Jiacheng Yu (4 papers)
  3. Yuhan Guo (10 papers)
  4. Jiayi Zhuang (2 papers)
  5. Yuchu Luo (4 papers)
  6. Xiaoru Yuan (12 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets