Papers
Topics
Authors
Recent
Search
2000 character limit reached

What-if Analysis for Business Professionals: Current Practices and Future Opportunities

Published 27 Dec 2022 in cs.HC and cs.DB | (2212.13643v4)

Abstract: What-if analysis (WIA) is essential for data-driven decision-making, allowing users to assess how changes in variables impact outcomes and explore alternative scenarios. Existing WIA research primarily supports the workflows of data scientists and analysts, and largely overlooks business professionals who engage in WIA through non-technical means. To bridge this gap, we conduct a two-part user study with 22 business professionals across marketing, sales, product, and operations roles. The first study examines their existing WIA practices, tools, and challenges. Findings reveal that business professionals perform many WIA techniques independently using rudimentary tools due to various constraints. We then implement representative WIA techniques in a visual analytics prototype and use it as a probe to conduct a follow-up study evaluating business professionals' practical use of the techniques. Results show that these techniques improve decision-making efficiency and confidence while underscoring the need for better data preparation, risk assessment, and domain knowledge integration support. Finally, we offer design recommendations to enhance future business analytics systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer, “Enterprise data analysis and visualization: An interview study,” TVCG, vol. 18, no. 12, pp. 2917–2926, 2012.
  2. S. Alspaugh, N. Zokaei, A. Liu, C. Jin, and M. A. Hearst, “Futzing and moseying: Interviews with professional data analysts on exploration practices,” TVCG, vol. 25, no. 1, p. 22–31, jan 2019.
  3. E. Kandogan, A. Balakrishnan, E. M. Haber, and J. S. Pierce, “From data to insight: work practices of analysts in the enterprise,” CGA, vol. 34, no. 5, pp. 42–50, 2014.
  4. A. Crisan and B. Fiore-Gartland, “Fits and starts: Enterprise use of automl and the role of humans in the loop,” in CHI, 2021, pp. 1–15.
  5. P. Mikalef, M. N. Giannakos, I. O. Pappas, and J. Krogstie, “The human side of big data: Understanding the skills of the data scientist in education and industry,” in EDUCON 2018.   IEEE, 2018, pp. 503–512.
  6. Ç. Demiralp, P. J. Haas, S. Parthasarathy, and T. Pedapati, “Foresight: Rapid data exploration through guideposts,” CoRR, vol. abs/1709.10513, 2017.
  7. Report, “Advanced Analytics Report, 2022 - 2030.”
  8. ——, “Data Visualization Statistics 2023.”
  9. J. D. Little, “Decision support systems for marketing managers,” JM, vol. 43, no. 3, pp. 9–26, 1979.
  10. W. S. Perkins and R. C. Rao, “The role of experience in information use and decision making by marketing managers,” JMR, vol. 27, no. 1, pp. 1–10, 1990.
  11. F. Bergeron, L. Raymond, S. Rivard, and M.-F. Gara, “Determinants of eis use: Testing a behavioral model,” Decision Support Systems, vol. 14, no. 2, pp. 131–146, 1995.
  12. F. Marx, J. H. Mayer, and R. Winter, “Six principles for redesigning executive information systems—findings of a survey and evaluation of a prototype,” ACM TMIS, vol. 2, no. 4, pp. 1–19, 2012.
  13. G. H. Van Bruggen, B. Wierenga et al., “Marketing decision making and decision support: Challenges and perspectives for successful marketing management support systems,” Foundations and Trends in Marketing, vol. 4, no. 4, pp. 209–332, 2010.
  14. O. Velcu-Laitinen and O. M. Yigitbasioglu, “The use of dashboards in performance management: Evidence from sales managers.” IJDAR, vol. 12, 2012.
  15. E. Dimara, H. Zhang, M. Tory, and S. Franconeri, “The unmet data visualization needs of decision makers within organizations,” TVCG, 2021.
  16. M. Jasim, E. Hoque, A. Sarvghad, and N. Mahyar, “Communitypulse: Facilitating community input analysis by surfacing hidden insights, reflections, and priorities,” in DIS, 2021, pp. 846–863.
  17. L. South, M. Schwab, N. Beauchamp, L. Wang, J. Wihbey, and M. A. Borkin, “Debatevis: Visualizing political debates for non-expert users,” in VIS.   IEEE, 2020, pp. 241–245.
  18. Y. Kang and J. Stasko, “Examining the use of a visual analytics system for sensemaking tasks: Case studies with domain experts,” TVCG, vol. 18, no. 12, pp. 2869–2878, 2012.
  19. J. Liu, N. Boukhelifa, and J. Eagan, “Making sense of data workers’ sense making practices,” in CHI Workshop, 2019, p. 4.
  20. L. R. Bartram, M. Correll, and M. K. Tory, “Untidy data: The unreasonable effectiveness of tables,” TVCG, vol. PP, pp. 1–1, 2021.
  21. H. A. Simon, “The new science of management decision.” 1960.
  22. A. Berisha-Shaqiri, “Management information system and decision-making,” AJIS, vol. 3, no. 2, p. 19, 2014.
  23. S. Gathani, M. Hulsebos, J. Gale, P. J. Haas, and Ç. Demiralp, “Augmenting decision making via interactive what-if analysis,” in CIDR 2022. [Online]. Available: https://www.cidrdb.org/cidr2022/papers/p49-gathani.pdf
  24. Y. Zhang, Q. V. Liao, and R. K. Bellamy, “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making,” in CFAT, 2020, pp. 295–305.
  25. Z. Zhang, S. Malik, S. Guo, J. Hoffswell, R. Rossi, F. Du, and E. Koh, “CODAS: Integrating Business Analytics and Report Authoring,” in EuroVA, J. Bernard and M. Angelini, Eds., 2022.
  26. G. Klein, B. Moon, and R. R. Hoffman, “Making sense of sensemaking 2: A macrocognitive model,” IEEE Intelligent Systems, vol. 21, no. 5, pp. 88–92, 2006.
  27. D. M. Russell, M. J. Stefik, P. Pirolli, and S. K. Card, “The cost structure of sensemaking,” in INTERACT and CHI, 1993, pp. 269–276.
  28. G. Klein, J. K. Phillips, E. L. Rall, and D. A. Peluso, “A data-frame theory of sensemaking,” in Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making, vol. 113, 2007.
  29. P. Pirolli and S. Card, “The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis,” in CIA, vol. 5, 2005, pp. 2–4.
  30. F. C. Lunenburg, “The decision making process.” in National Forum of Educational Administration & Supervision Journal, vol. 27, no. 4, 2010.
  31. E. Bruch and F. Feinberg, “Decision-making processes in social contexts,” Annual review of sociology, vol. 43, pp. 207–227, 2017.
  32. C. Chen, “An information-theoretic view of visual analytics,” CGA, vol. 28, no. 1, pp. 18–23, 2008.
  33. N. E. Eric Newburger, “Visualization according to statisticians: An interview study on the role of visualization for inferential statistics,” 2023.
  34. W. Noonpakdee, T. Khunkornsiri, A. Phothichai, and K. Danaisawat, “A framework for analyzing and developing dashboard templates for small and medium enterprises,” in ICIEA.   IEEE, 2018, pp. 479–483.
  35. P. Bera, “How colors in business dashboards affect users’ decision making,” CACM, vol. 59, no. 4, pp. 50–57, 2016.
  36. E. Dimara and J. Stasko, “A critical reflection on visualization research: Where do decision making tasks hide?” TVCG, vol. 28, no. 1, pp. 1128–1138, 2021.
  37. A. Sarikaya, M. Correll, L. Bartram, M. Tory, and D. Fisher, “What do we talk about when we talk about dashboards?” TVCG, vol. 25, no. 1, pp. 682–692, 2018.
  38. M. Tory, L. Bartram, B. Fiore-Gartland, and A. Crisan, “Finding their data voice: Practices and challenges of dashboard users,” CGA, 2021.
  39. A. Crisan and M. Correll, “User ex machina: Simulation as a design probe in human-in-the-loop text analytics,” in CHI, 2021, pp. 1–16.
  40. D. Honeycutt, M. Nourani, and E. Ragan, “Soliciting human-in-the-loop user feedback for interactive machine learning reduces user trust and impressions of model accuracy,” in AAAI, vol. 8, 2020, pp. 63–72.
  41. D. Wang, J. D. Weisz, M. Muller, P. Ram, W. Geyer, C. Dugan, Y. Tausczik, H. Samulowitz, and A. Gray, “Human-ai collaboration in data science: Exploring data scientists’ perceptions of automated ai,” Proc. ACM Hum.-Comput. Interact., vol. 3, no. CSCW, nov 2019.
  42. F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W. Wortman Vaughan, and H. Wallach, “Manipulating and measuring model interpretability,” in CHI, 2021, pp. 1–52.
  43. Tableau, “What is tableau business science?” 2021. [Online]. Available: https://www.tableau.com/about/blog/2021/3/what-is-tableau-business-science
  44. ——, “Einstein discovery,” 2021. [Online]. Available: https://www.tableau.com/products/add-ons/einstein-discovery
  45. SAS, “Sas visual analytics,” 2021. [Online]. Available: https://www.sas.com/en_us/software/visual-analytics.html
  46. Microsoft, “Solver: Microsoft Excel Documentation,” 2021.
  47. ——, “Goal Seek: Microsoft Excel Documentation,” 2021.
  48. E. Jun, “Empowering domain experts to author valid statistical analyses,” in UIST, 2022, pp. 1–5.
  49. S. Gathani, Z. Liu, P. J. Haas, and Ç. Demiralp, “Online Form for Participant Recruitment.”
  50. “User interviews,” 2021. [Online]. Available: https://www.userinterviews.com
  51. S. Gathani, Z. Liu, P. J. Haas, and Ç. Demiralp, “Supplementary Material.”
  52. Kaggle, “Kaggle Datasets.”
  53. A. Rappaport, “Sensitivity analysis in decision making,” The Accounting Review, vol. 42, no. 3, pp. 441–456, 1967.
  54. N. H. Lurie and C. H. Mason, “Visual representation: Implications for decision making,” JM, vol. 71, no. 1, pp. 160–177, 2007.
  55. S. Gathani, Z. Liu, P. J. Haas, and Ç. Demiralp, “Post Study Questionnaire.”
  56. H. Elhamdadi, A. Gaba, Y.-S. Kim, and C. Xiong, “How do we measure trust in visual data communication?” in BELIV, 2022, pp. 85–92.
  57. M. Langer, T. Hunsicker, T. Feldkamp, C. J. König, and N. Grgić-Hlača, ““look! it’sa computer program! it’s an algorithm! it’s ai!”: does terminology affect human perceptions and evaluations of algorithmic decision-making systems?” in CHI, 2022, pp. 1–28.
  58. H. Spitzeck and E. G. Hansen, “Stakeholder governance: how stakeholders influence corporate decision making,” Corporate Governance: The international journal of business in society, vol. 10, no. 4, pp. 378–391, 2010.
  59. A. Schreiber, L. von Kurnatowski, A. Meinecke, and C. de Boer, “An interactive dashboard for visualizing the provenance of software development processes,” in VISSOFT.   IEEE, 2021, pp. 100–104.
  60. H. S. Packer, A. Chapman, and L. Carr, “Github2prov: Provenance for supporting software project management.” in TaPP, 2019.
  61. Z. Cutler, K. Gadhave, and A. Lex, “Trrack: A library for provenance-tracking in web-based visualizations,” in VIS, 2020, pp. 116–120.
  62. K. Madanagopal, E. D. Ragan, and P. Benjamin, “Analytic provenance in practice: The role of provenance in real-world visualization and data analysis environments,” CGA, vol. 39, no. 6, pp. 30–45, 2019.
  63. E. D. Ragan, A. Endert, J. Sanyal, and J. Chen, “Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes,” TVCG, vol. 22, no. 1, pp. 31–40, 2015.
  64. M. H. Namaki, A. Floratou, F. Psallidas, S. Krishnan, A. Agrawal, Y. Wu, Y. Zhu, and M. Weimer, “Vamsa: Automated provenance tracking in data science scripts,” in SIGKDD, 2020, pp. 1542–1551.
  65. R. Souza, L. G. Azevedo, V. Lourenço, E. Soares, R. Thiago, R. Brandão, D. Civitarese, E. Vital Brazil, M. Moreno, P. Valduriez et al., “Workflow provenance in the lifecycle of scientific machine learning,” CCPE, vol. 34, no. 14, p. e6544, 2022.
  66. A. Kumar, R. McCann, J. Naughton, and J. M. Patel, “Model selection management systems: The next frontier of advanced analytics,” SIGMOD, vol. 44, no. 4, pp. 17–22, 2016.
  67. E. Jun, M. Daum, J. Roesch, S. Chasins, E. Berger, R. Just, and K. Reinecke, “Tea: A high-level language and runtime system for automating statistical analysis,” in UIST, 2019, pp. 591–603.
  68. E. Jun, M. Birchfield, N. De Moura, J. Heer, and R. Just, “Hypothesis formalization: Empirical findings, software limitations, and design implications,” TOCHI, vol. 29, no. 1, pp. 1–28, 2022.
  69. E. Jun, A. Seo, J. Heer, and R. Just, “Tisane: Authoring statistical models via formal reasoning from conceptual and data relationships,” in CHI, 2022, pp. 1–16.
  70. K. Gu, E. Jun, and T. Althoff, “Understanding and supporting debugging workflows in multiverse analysis,” in CHI, 2023, pp. 1–19.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up