Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing (2401.10880v1)

Published 19 Jan 2024 in cs.HC

Abstract: Users often rely on GUIs to edit and interact with visualizations - a daunting task due to the large space of editing options. As a result, users are either overwhelmed by a complex UI or constrained by a custom UI with a tailored, fixed subset of options with limited editing flexibility. Natural Language Interfaces (NLIs) are emerging as a feasible alternative for users to specify edits. However, NLIs forgo the advantages of traditional GUI: the ability to explore and repeat edits and see instant visual feedback. We introduce DynaVis, which blends natural language and dynamically synthesized UI widgets. As the user describes an editing task in natural language, DynaVis performs the edit and synthesizes a persistent widget that the user can interact with to make further modifications. Study participants (n=24) preferred DynaVis over the NLI-only interface citing ease of further edits and editing confidence due to immediate visual feedback.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. 2020. OpenAI API. https://openai.com/blog/openai-api. Accessed: 2023-9-13.
  2. 2023. Apache OpenNLP. https://opennlp.apache.org/. Accessed: 2023-9-13.
  3. 2023. Cloud Natural Language. https://cloud.google.com/natural-language. Accessed: 2023-9-13.
  4. 2023. D3 by Observable. https://d3js.org/. Accessed: 2023-9-14.
  5. 2023. ggplot2. https://ggplot2.tidyverse.org/. Accessed: 2023-9-14.
  6. 2023a. Microsoft PowerBI. https://powerbi.microsoft.com/en-us/. Accessed: 2023-9-14.
  7. 2023. npm: html-react-parser. https://www.npmjs.com/package/html-react-parser. Accessed: 2023-12-6.
  8. 2023. pandas documentation — pandas 2.1.0 documentation. https://pandas.pydata.org/docs/index.html. Accessed: 2023-9-14.
  9. 2023. spaCy - Industrial-strength Natural Language Processing in Python. https://spacy.io/. Accessed: 2023-9-13.
  10. 2023. Tableau: Business Intelligence and Analytics Software. https://www.tableau.com/. Accessed: 2023-9-14.
  11. 2023. The Two UX Gulfs: Evaluation and Execution. https://www.nngroup.com/articles/two-ux-gulfs-evaluation-execution/. Accessed: 2023-9-14.
  12. 2023b. Use natural language to explore data with Power BI Q&A - Power BI. https://learn.microsoft.com/en-us/power-bi/natural-language/q-and-a-intro. Accessed: 2023-9-13.
  13. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOPSLA1, Article 78 (apr 2023), 27 pages. https://doi.org/10.1145/3586030
  14. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics 7 (2019), 49–72.
  15. Raluca Budiu. 2014. Memory Recognition and Recall in User Interfaces. https://www.nngroup.com/articles/recognition-and-recall/. Accessed: 2023-9-12.
  16. Type-directed synthesis of visualizations from natural language queries. Proceedings of the ACM on Programming Languages 6, OOPSLA2 (2022), 532–559.
  17. Yiru Chen. 2020. Monte carlo tree search for generating interactive data analysis interfaces. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 2837–2839.
  18. NL2INTERFACE: Interactive Visualization Interface Generation from Natural Language Queries. ArXiv abs/2209.08834 (2022). https://api.semanticscholar.org/CorpusID:252367337
  19. Yiru Chen and Eugene Wu. 2022. Pi2: End-to-end interactive visualization interface generation from queries. In Proceedings of the 2022 International Conference on Management of Data. 1711–1725.
  20. Vizdom: interactive analytics through pen and touch. Proceedings of the VLDB Endowment 8, 12 (2015), 2024–2027.
  21. Analyza: Exploring data with conversation. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 493–504.
  22. Victor Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. (6 March 2023). arXiv:2303.02927 [cs.AI]
  23. Victor Dibia and Çağatay Demiralp. 2019. Data2vis: Automatic generation of data visualizations using sequence-to-sequence recurrent neural networks. IEEE computer graphics and applications 39, 5 (2019), 33–46.
  24. DIY assistant: a multi-modal end-user programmable virtual assistant. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 312–327.
  25. Krzysztof Gajos and Daniel S Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces. 93–100.
  26. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 231–240.
  27. Datatone: Managing ambiguity in natural language interfaces for data visualization. In Proceedings of the 28th annual acm symposium on user interface software & technology. 489–500.
  28. Generalized selection via interactive query relaxation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 959–968.
  29. You can’t always sketch what you want: Understanding sensemaking in visual query systems. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1267–1277.
  30. Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations. In Proceedings of the 32nd annual ACM symposium on user interface software and technology. 577–589.
  31. Data illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
  32. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. arXiv preprint cs/0205028 (2002).
  33. Synthesis of web layouts from examples. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 651–663.
  34. Deepeye: Creating good data visualizations by keyword search. In Proceedings of the 2018 International Conference on Management of Data. 1733–1736.
  35. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. 55–60.
  36. Ruhaab Markas. [n. d.]. Ask Data: Simplifying analytics with natural language. https://www.tableau.com/blog/ask-data-simplifying-analytics-natural-language-98655. Accessed: 2023-9-13.
  37. Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco. IEEE transactions on visualization and computer graphics 25, 1 (2018), 438–448.
  38. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 369–379.
  39. Personal universal controllers: controlling complex appliances with GUIs and speech. In CHI’03 Extended Abstracts on Human Factors in Computing Systems. 624–625.
  40. UNIFORM: automatically generating consistent remote control user interfaces. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 611–620.
  41. Huddle: automatically generating interfaces for systems of multiple connected appliances. In Proceedings of the 19th annual ACM symposium on User interface software and technology. 279–288.
  42. Jakob Nielsen. 2020. 10 Usability Heuristics for User Interface Design. https://www.nngroup.com/articles/ten-usability-heuristics/. Accessed: 2023-9-13.
  43. Donald Norman. 1986. User centered system design. New perspectives on human-computer interaction (1986).
  44. Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors 39, 2 (1997), 230–253.
  45. Charticulator: Interactive construction of bespoke chart layouts. IEEE transactions on visualization and computer graphics 25, 1 (2018), 789–799.
  46. Arvind Satyanarayan and Jeffrey Heer. 2014. Lyra: An interactive visualization design environment. In Computer graphics forum, Vol. 33. Wiley Online Library, 351–360.
  47. Vega-lite: A grammar of interactive graphics. IEEE transactions on visualization and computer graphics 23, 1 (2016), 341–350.
  48. Eviza: A natural language interface for visual analysis. In Proceedings of the 29th annual symposium on user interface software and technology. 365–377.
  49. Towards natural language interfaces for data visualization: A survey. IEEE transactions on visualization and computer graphics (2022).
  50. Shapesearch: A flexible and efficient system for shape-based exploration of trendlines. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 51–65.
  51. Arjun Srinivasan and Vidya Setlur. 2023. BOLT: A Natural Language Interface for Dashboard Authoring. (2023).
  52. Arjun Srinivasan and John Stasko. 2017. Natural language interfaces for data analysis with visualization: Considering what has and could be asked. In Proceedings of the Eurographics/IEEE VGTC conference on visualization: Short papers. 55–59.
  53. Priyan Vaithilingam and Philip J Guo. 2019. Bespoke: Interactively synthesizing custom GUIs from command-line applications by demonstration. In Proceedings of the 32nd annual ACM symposium on user interface software and technology. 563–576.
  54. Mavo: Creating Interactive Data-Driven Web Applications by Authoring HTML. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 483–496. https://doi.org/10.1145/2984511.2984551
  55. Towards natural language-based visualization authoring. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2022), 1222–1232.
  56. Leland Wilkinson. 2005. The Grammar of Graphics, Second Edition. Springer.
  57. Voyager 2: Augmenting visual analysis with partial view specifications. In Proceedings of the 2017 chi conference on human factors in computing systems. 2648–2659.
  58. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine 13, 3 (2018), 55–75.
  59. Bowen Yu and Cláudio T Silva. 2019. FlowSense: A natural language interface for visual data exploration within a dataflow system. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1–11.
  60. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
  61. Panoramicdata: Data analysis through pen & touch. IEEE transactions on visualization and computer graphics 20, 12 (2014), 2112–2121.
  62. Sato: Contextual semantic type detection in tables. arXiv preprint arXiv:1911.06311 (2019).
  63. Precision interfaces for different modalities. In Proceedings of the 2018 International Conference on Management of Data. 1777–1780.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Priyan Vaithilingam (7 papers)
  2. Elena L. Glassman (19 papers)
  3. Jeevana Priya Inala (18 papers)
  4. Chenglong Wang (80 papers)
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now