Beyond Generating Code: Evaluating GPT on a Data Visualization Course (2306.02914v3)

Abstract: This paper presents an empirical evaluation of the performance of the Generative Pre-trained Transformer (GPT) model in Harvard's CS171 data visualization course. While previous studies have focused on GPT's ability to generate code for visualizations, this study goes beyond code generation to evaluate GPT's abilities in various visualization tasks, such as data interpretation, visualization design, visual data exploration, and insight communication. The evaluation utilized GPT-3.5 and GPT-4 to complete assignments of CS171, and included a quantitative assessment based on the established course rubrics, a qualitative analysis informed by the feedback of three experienced graders, and an exploratory study of GPT's capabilities in completing border visualization tasks. Findings show that GPT-4 scored 80% on quizzes and homework, and TFs could distinguish between GPT- and human-generated homework with 70% accuracy. The study also demonstrates GPT's potential in completing various visualization tasks, such as data cleanup, interaction with visualizations, and insight communication. The paper concludes by discussing the strengths and limitations of GPT in data visualization, potential avenues for incorporating GPT in broader visualization tasks, and the need to redesign visualization education.

• The paper evaluates GPT-3.5 and GPT-4 by assessing their performance on data visualization tasks using Harvard’s CS171 course rubrics.
• The paper reveals that while GPT-4 can interact with visualizations and generate code effectively, it struggles with multi-file reasoning and image-based inputs.
• The paper suggests that integrating AI into visualization education could drive curriculum redesigns that focus on creative, human-centric problem solving.

Background and Introduction

Generative Pre-trained Transformers, or GPT models, have garnered significant attention for their ability to produce human-like text across a wide range of natural language processing tasks. Developed by OpenAI, variants such as GPT-3 and GPT-4 have been put to the test, demonstrating their prowess in generating code, passing professional exams, and even hinting at early forms of artificial general intelligence. In the domain of data visualization, they were mostly scrutinized for their code production abilities, but have not been extensively evaluated in other visualization-related tasks, such as design or data insight communication.

Exploring GPT's Role in Data Visualization

In a unique paper conducted on Harvard’s CS171 data visualization course, researchers assessed the performance of GPT-3.5 and GPT-4 beyond traditional code generation. They examined the model's ability in interpreting data, designing visualizations, exploring visual data, and communicating insights effectively. Utilizing established course rubrics and feedback from experienced graders, they conducted a quantitative and qualitative analysis on GPT's outputs. Both GPT-3.5 and GPT-4 completed the course's quizzes and homework assignments, scoring notable grades. Interestingly, graders could distinguish the AI’s work with relative accuracy, although they were misled at times, mistaking AI-generated work for human submissions in some instances.

Strengths and Potential for GPT in Data Visualization

The performance of the models unearthed GPT's potential for executing a variety of visualization tasks such as data cleanup, working with visualization libraries, and interactive elements. For instance, GPT-4 demonstrated an ability to interact with visualizations, dispatching JavaScript events, and offered insights into complex visualizations. Despite these strengths, researchers also observed limitations in GPT's reasoning when dealing with multiple code files, or when input data included images, as the current version tested didn't support image processing.

Challenges, Ethical Considerations, and Educational Implications

The paper did not only highlight the technical abilities but also underscored certain concerns. Researchers noted instances of "hallucination," where GPT would make assumptions about images it couldn't actually see, and "ethical constraints" where it refused tasks it deemed as cheating. As part of a broader discussion on the implications of utilizing GPT and similar models, the paper stresses examining their output critically, especially considering these models might reflect or amplify biases present in their training data.

An important subject triggered by the findings is the potential need to redesign educational approaches in visualization. The results suggest an opportunity to re-evaluate how visualization is taught, moving away from tasks AI models can easily perform towards those requiring unique human judgment and creativity. Moreover, instructors are encouraged to explore AI incorporation into the curriculum, creating a collaborative learning environment that both leverages and scrutinizes these AI tools.

The investigation presented in the paper invites numerous avenues for further research in GPT's role within the field of data visualization, pushing forward the discussion about AI's expanding reach in academia and professional settings.