InsigHTable: Insight-driven Hierarchical Table Visualization with Reinforcement Learning (2405.17229v1)
Abstract: Embedding visual representations within original hierarchical tables can mitigate additional cognitive load stemming from the division of users' attention. The created hierarchical table visualizations can help users understand and explore complex data with multi-level attributes. However, because of many options available for transforming hierarchical tables and selecting subsets for embedding, the design space of hierarchical table visualizations becomes vast, and the construction process turns out to be tedious, hindering users from constructing hierarchical table visualizations with many data insights efficiently. We propose InsigHTable, a mixed-initiative and insight-driven hierarchical table transformation and visualization system. We first define data insights within hierarchical tables, which consider the hierarchical structure in the table headers. Since hierarchical table visualization construction is a sequential decision-making process, InsigHTable integrates a deep reinforcement learning framework incorporating an auxiliary rewards mechanism. This mechanism addresses the challenge of sparse rewards in constructing hierarchical table visualizations. Within the deep reinforcement learning framework, the agent continuously optimizes its decision-making process to create hierarchical table visualizations to uncover more insights by collaborating with analysts. We demonstrate the usability and effectiveness of InsigHTable through two case studies and sets of experiments. The results validate the effectiveness of the deep reinforcement learning framework and show that InsigHTable can facilitate users to construct hierarchical table visualizations and understand underlying data insights.
- C. Scaffidi, M. Shaw, and B. Myers, “Estimating the numbers of end users and end user programmers,” in Proc. IEEE Symp. Visual Languages and Human-Centric Computing (VL/HCC), 2005, pp. 207–214.
- W. Dou, S. Han, L. Xu, D. Zhang, and J. Wei, “Expandable group identification in spreadsheets,” in Proc. ACM/IEEE Int. Conf. Automated Software Engineering (ASE), 2018, pp. 498–508.
- K. Furmanova, S. Gratzl, H. Stitz, T. Zichner, M. Jaresova, A. Lex, and M. Streit, “Taggle: Scalable visualization of tabular data through aggregation,” Information Visualisation, vol. 19, no. 2, pp. 114–136, 2019.
- Q. Yang, Y. Cao, H. Li, and P. Luo, “Numerical formula recognition from tables,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2021, pp. 1986–1996.
- Q. Yang, Y. Cao, and P. Luo, “Numerical tuple extraction from tables with pre-training,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2022, p. 2233–2241.
- L. Du, F. Gao, X. Chen, R. Jia, J. Wang, J. Zhang, S. Han, and D. Zhang, “Tabularnet: A neural network architecture for understanding semantic structures of tabular data,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2021, pp. 322–331.
- G. Li, R. Li, Z. Wang, C. H. Liu, M. Lu, and G. Wang, “HiTailor: Interactive transformation and visualization for hierarchical tabular data,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 1, pp. 139–148, 2022.
- H.-J. Schulz, “Treevis. net: A tree visualization reference,” IEEE Computer Graphics and Applications, vol. 31, no. 6, pp. 11–15, 2011.
- G. Li, M. Tian, Q. Xu, M. J. McGuffin, and X. Yuan, “GoTree: A grammar of tree visualizations,” in Proc. ACM Conf. Human Factors in Computing Systems (CHI), 2020, p. 1–13.
- G. Li, Y. Zhang, Y. Dong, J. Liang, J. Zhang, J. Wang, M. J. Mcguffin, and X. Yuan, “BarcodeTree: Scalable comparison of multiple hierarchies,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, pp. 1022–1032, 2020.
- Z. Wang, H. Dong, R. Jia, J. Li, Z. Fu, S. Han, and D. Zhang, “TUTA: Tree-based transformers for generally structured table pre-training,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2021, pp. 1780–1790.
- Z. Cheng, H. Dong, R. Jia, J. Guo, Y. Gao, S. Han, J.-G. Lou, and D. Zhang, “HiTab: A hierarchical table dataset for question answering and natural language generation,” in Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2022, pp. 1094–1110.
- G. Li, R. Li, Y. Feng, Y. Zhang, Y. Luo, and C. H. Liu, “CoInsight: Visual storytelling for hierarchical tables with connected insights,” IEEE Transactions on Visualization and Computer Graphics, pp. 1–11, 2024.
- P. Ginns, “Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects,” Learning and Instruction, vol. 16, no. 6, pp. 511–525, 2006.
- A. Paul and S. John, “The split-attention principle in multimedia learning,” The Cambridge Handbook of Multimedia Learning, vol. 2, pp. 135–146, 2005.
- J. Sweller, “Implications of cognitive load theory for multimedia learning,” The Cambridge Handbook of Multimedia Learning, vol. 3, no. 2, pp. 19–30, 2005.
- J. Sweller, J. J. Van Merrienboer, and F. G. Paas, “Cognitive architecture and instructional design,” Educational Psychology Review, vol. 10, no. 3, pp. 251–296, 1998.
- D. Park, S. M. Drucker, R. Fernandez, and N. Elmqvist, “ATOM: A grammar for unit visualizations,” IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 12, pp. 3032–3043, 2018.
- S. Gratzl, A. Lex, N. Gehlenborg, H. Pfister, and M. Streit, “LineUp: Visual analysis of multi-attribute rankings,” IEEE Transactions on Visualization and Computer Graphics, vol. 19, no. 12, pp. 2277–2286, 2013.
- Z. Gu, R. Eils, and M. Schlesner, “Complex heatmaps reveal patterns and correlations in multidimensional genomic data,” Bioinformatics, vol. 32, no. 18, pp. 2847–2849, 2016.
- C. Perin, P. Dragicevic, and J. Fekete, “Revisiting Bertin Matrices: New interactions for crafting tabular visualizations,” IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 12, pp. 2082–2091, 2014.
- R. Rao and S. K. Card, “The table lens: merging graphical and symbolic representations in an interactive focus + context visualization for tabular information,” in Proc. ACM Conf. Human Factors in Computing Systems (CHI), 1994, pp. 318–322.
- A. Lex, H.-J. Schulz, M. Streit, C. Partl, and D. Schmalstieg, “Visbricks: Multiform visualization of large, inhomogeneous data,” IEEE Transactions on Visualization and Computer Graphics, vol. 17, no. 12, pp. 2291–2300, 2011.
- B. Tang, S. Han, M. L. Yiu, R. Ding, and D. Zhang, “Extracting top-k insights from multi-dimensional data,” in Proc. Int. ACM Conf. Management of Data (SIGMOD), 2017, pp. 1509–1524.
- R. Ding, S. Han, Y. Xu, H. Zhang, and D. Zhang, “Quickinsights: Quick and automatic discovery of insights from multi-dimensional data,” in Proc. Int. ACM Conf. Management of Data (SIGMOD), 2019, pp. 317–332.
- P. Ma, R. Ding, S. Han, and D. Zhang, “Metainsight: Automatic discovery of structured knowledge for exploratory data analysis,” in Proc. Int. ACM Conf. Management of Data (SIGMOD), 2021, pp. 1262–1274.
- Q. Lin, W. Ke, J.-G. Lou, H. Zhang, K. Sui, Y. Xu, Z. Zhou, B. Qiao, and D. Zhang, “BigIN4: Instant, interactive insight identification for multi-dimensional big data,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2018, pp. 547–555.
- C. Demiralp, P. J. Haas, S. Parthasarathy, and T. Pedapati, “Foresight: Recommending visual insights,” Proc. Int. Conf. Very Large Data Bases (VLDB), vol. 10, no. 12, pp. 1937–1940, 2017.
- J. Peng, W. Wu, B. Lockhart, S. Bian, J. N. Yan, L. Xu, Z. Chi, J. M. Rzeszotarski, and J. Wang, “Dataprep. eda: task-centric exploratory data analysis for statistical modeling in python,” in Proc. Int. ACM Conf. Management of Data (SIGMOD), 2021, pp. 2271–2280.
- Y. Wang, Z. Sun, H. Zhang, W. Cui, K. Xu, X. Ma, and D. Zhang, “Datashot: Automatic generation of fact sheets from tabular data,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, pp. 895–905, 2019.
- D. Shi, X. Xu, F. Sun, Y. Shi, and N. Cao, “Calliope: Automatic visual data story generation from a spreadsheet,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, pp. 453–463, 2020.
- D. Deng, A. Wu, H. Qu, and Y. Wu, “Dashbot: Insight-driven dashboard generation based on deep reinforcement learning,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 1, pp. 690–700, 2022.
- O. Bar El, T. Milo, and A. Somech, “Automatically generating data exploration sessions using deep reinforcement learning,” in Proc. Int. ACM Conf. Management of Data (SIGMOD), 2020, pp. 1527–1537.
- Z. Cui, S. K. Badam, M. A. Yalçin, and N. Elmqvist, “DataSite: Proactive visual data exploration with computation of insight-based recommendations,” IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 2, pp. 251–267, 2019.
- A. Srinivasan, S. M. Drucker, A. Endert, and J. Stasko, “Augmenting visualizations with interactive data facts to facilitate interpretation and communication,” IEEE Transactions on Visualization and Computer Graphics, vol. 25, no. 1, pp. 672–681, 2018.
- C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992.
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
- R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 12, pp. 1057–1063, 1999.
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016.
- O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, and P. Georgiev, “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, no. 7782, pp. 350–354, 2019.
- J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, and D. de Las Casas, “Magnetic control of tokamak plasmas through deep reinforcement learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022.
- A. Wu, W. Tong, T. Dwyer, B. Lee, P. Isenberg, and H. Qu, “Mobilevisfixer: Tailoring web visualizations for mobile phones leveraging an explainable reinforcement learning framework,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, pp. 464–474, 2020.
- Q. Chen, F. Sun, X. Xu, Z. Chen, J. Wang, and N. Cao, “Vizlinter: A linter and fixer framework for data visualization,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 1, pp. 206–216, 2021.
- M. El-Assady, R. Sevastjanova, F. Sperrle, D. Keim, and C. Collins, “Progressive learning of topic modeling parameters: A visual analytics framework,” IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 382–391, 2017.
- J.-F. Kassel and M. Rohs, “Online learning of visualization preferences through dueling bandits for enhancing visualization recommendations.” in Proc. Eurographics / IEEE VGTC Conference on Visualization (EuroVis), 2019, pp. 85–89.
- R. Hu, B. Chen, J. Xu, O. Van Kaick, O. Deussen, and H. Huang, “Shape-driven coordinate ordering for star glyph sets via reinforcement learning,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 6, pp. 3034–3047, 2021.
- M. Zhou, Q. Li, X. He, Y. Li, Y. Liu, W. Ji, S. Han, Y. Chen, D. Jiang, and D. Zhang, “Table2Charts: recommending charts by learning shared table representations,” in Proc. ACM Conf. Knowledge Discovery and Data Mining (SIGKDD), 2021, pp. 2389–2399.
- T. Tang, R. Li, X. Wu, S. Liu, J. Knittel, S. Koch, T. Ertl, L. Yu, P. Ren, and Y. Wu, “Plotthread: Creating expressive storyline visualizations using reinforcement learning,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, pp. 294–303, 2020.
- I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio, “Neural combinatorial optimization with reinforcement learning,” in Proc. Int. Conf. Learning Representation (ICLR), 2016.
- O. Vinyals, M. Fortunato, and N. Jaitly, “Pointer networks,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 28, pp. 2692–2700, 2015.
- Y. Shi, B. Chen, Y. Chen, Z. Jin, K. Xu, X. Jiao, T. Gao, and N. Cao, “Supporting guided exploratory visual analysis on time series data with reinforcement learning,” IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 1, pp. 1172–1182, 2024.
- R. Amar, J. Eagan, and J. Stasko, “Low-level components of analytic activity in information visualization,” in Proc. IEEE Symp. Information Visualization (InfoVis), 2005, pp. 111–117.
- H.-J. Schulz, T. Nocke, M. Heitzler, and H. Schumann, “A design space of visualization tasks,” IEEE Transactions on Visualization and Computer Graphics, vol. 19, no. 12, pp. 2366–2375, 2013.
- M. Tory and T. Moller, “Rethinking visualization: A high-level taxonomy,” in Proc. IEEE Symp. Information Visualization (InfoVis), 2004, pp. 151–158.
- R. Bellman, “A markovian decision process,” Journal of mathematics and mechanics, pp. 679–684, 1957.
- K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo, “VizML: A machine learning approach to visualization recommendation,” in Proc. ACM Conf. Human Factors in Computing Systems (CHI), 2019, pp. 1–12.
- T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in Proc. Int. Conf. Learning Representation (ICLR), 2017.
- J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
- Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” in Proc. Int. Conf. Learning Representation (ICLR), 2019, pp. 1–17.
- E. R. Chen, Z.-W. Hong, J. Pajarinen, and P. Agrawal, “Redeeming intrinsic rewards via constrained optimization,” in Proc. Advances in Neural Information Processing Systems (NeurIPS).
- E. Horvitz, “Principles of mixed-initiative user interfaces,” in Proc. ACM Conf. Human Factors in Computing Systems (CHI), 1999, pp. 159–166.
- K. Wongsuphasawat, D. Moritz, A. Anand, J. Mackinlay, B. Howe, and J. Heer, “Voyager: Exploratory analysis via faceted browsing of visualization recommendations,” IEEE Transactions on Visualization and Computer Graphics, vol. 22, no. 1, pp. 649–658, 2015.
- K. Wongsuphasawat, Z. Qu, D. Moritz, R. Chang, F. Ouk, A. Anand, J. Mackinlay, B. Howe, and J. Heer, “Voyager 2: Augmenting visual analysis with partial view specifications,” in Proc. ACM Conf. Human Factors in Computing Systems (CHI), 2017, pp. 2648–2659.
- A. W. Services, “Amazon quicksight,” https://aws.amazon.com/quicksight.
- G. Li and X. Yuan, “GoTreeScape: Navigate and explore the tree visualization design space,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 12, pp. 5451–5467, 2023.
- G. Li, M. Tian, Q. Xu, M. J. McGuffin, and X. Yuan, “Tree Illustrator: Interactive construction of tree visualizations,” in Extended Abstacts of ACM Conf. Human Factors in Computing Systems, ser. CHI EA ’20, 2020, pp. 1–4.
- M. Lu, J. Liang, Y. Zhang, G. Li, S. Chen, Z. Li, and X. Yuan, “Interaction+: Interaction enhancement for web-based visualizations,” in Proc. IEEE Pacific Visualization Symposium (PacificVis), 2017, pp. 61–70.
- J. Wang, L. Gou, H.-W. Shen, and H. Yang, “Dqnviz: A visual analytics approach to understand deep q-networks,” IEEE Transactions on Visualization and Computer Graphics, vol. 25, no. 1, pp. 288–298, 2018.
- Y. Metz, U. Schlegel, D. Seebacher, M. El-Assady, and D. A. Keim, “A comprehensive workflow for effective imitation and reinforcement learning with visual analytics,” in Proc. Int. EuroVis Workshop on Visual Analytics (EuroVA), 2022, pp. 19–23.
- A. Mishra, U. Soni, J. Huang, and C. Bryan, “Why? why not? when? visual explanations of agent behaviour in reinforcement learning,” in Proc. IEEE Pacific Visualization Symposium (PacificVis), 2022, pp. 111–120.
- W. Yang, M. Liu, Z. Wang, and S. Liu, “Foundation models meet visualizations: Challenges and opportunities,” Computational Visual Media, vol. 10, pp. 399–424, 2023.
- G. Li, X. Wang, G. Aodeng, S. Zheng, Y. Zhang, C. Ou, S. Wang, and C. H. Liu, “Visualization generation with large language models: An evaluation,” 2024, .