ChatBCG: Can AI Read Your Slide Deck? (2407.12875v1)

Abstract: Multimodal models like GPT4o and Gemini Flash are exceptional at inference and summarization tasks, which approach human-level in performance. However, we find that these models underperform compared to humans when asked to do very specific 'reading and estimation' tasks, particularly in the context of visual charts in business decks. This paper evaluates the accuracy of GPT 4o and Gemini Flash-1.5 in answering straightforward questions about data on labeled charts (where data is clearly annotated on the graphs), and unlabeled charts (where data is not clearly annotated and has to be inferred from the X and Y axis). We conclude that these models aren't currently capable of reading a deck accurately end-to-end if it contains any complex or unlabeled charts. Even if a user created a deck of only labeled charts, the model would only be able to read 7-8 out of 15 labeled charts perfectly end-to-end. For full list of slide deck figures visit https://www.repromptai.com/chat_bcg

• The paper demonstrates that GPT-4o and Gemini Flash-1.5 show error rates of 14-16% on labeled charts and up to 83% on unlabeled charts, far exceeding human performance.
• It employs rigorous metrics, including Match Rate, Mean Absolute Error, and Mean Absolute Percentage Error, across 31 chart types to assess model accuracy.
• The findings emphasize the need for improved pre-training and human oversight to enhance AI reliability in interpreting complex business data.

Chat BCG: Can AI Read Your Slide Deck?

The paper "Chat BCG: Can AI Read Your Slide Deck?" by Nikita Singh, Rob Balian, and Lukas Martinelli provides a critical evaluation of the capabilities of multimodal LLMs, specifically GPT-4o and Gemini Flash-1.5, in interpreting data from visual charts commonly found in business presentations. This research is key to understanding the limitations and potential of these models in practical business applications where accurate data interpretation is crucial.

Key Findings

The paper examines the performance of the models on two types of tasks: interpreting labeled charts, where data points are explicitly marked, and unlabeled charts, which require estimation based on the axes. The measurement involves assessing the models' accuracy in reading and estimating data points directly from these charts.

Labeled Charts

• Error Rates: Both GPT-4o and Gemini Flash-1.5 exhibit error rates of 16% and 14% respectively in interpreting labeled charts. This is significantly higher compared to the human error rate of under 5%.
• Error Patterns: The predominant errors involve misreading numbers, such as mistaking '3' for '8', and mislabeling positive numbers as negative. Neither model consistently outperforms the other across various types of labeled charts.
• Error Ranges: The range of errors varies widely, particularly with charts that contain multiple figures. For example, in more complex charts like stacked charts and waterfall charts, errors can be as substantial as misreading '2015' as '2009'.

Unlabeled Charts

• Error Rates: Error rates for unlabeled charts are alarmingly high, with rates reaching 79% for Gemini Flash-1.5 and 83% for GPT-4o. The average deviations from the correct values are 53% and 55% respectively, compared to 10-20% for humans.
• Error Magnitudes: The errors in estimation tasks often result in substantial deviations, indicating that these models frequently misread labels or apply incorrect estimations.

Methodology

The paper involved analyzing 31 different charts, split between 15 labeled and 16 unlabeled ones. The questions poised to the models aimed at:

• Identifying specific data points.
• Identifying the largest or smallest data points.
• Counting the number of data points.

Performance was measured using the Match Rate for labeled charts and Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) for unlabeled charts.

Practical and Theoretical Implications

The research highlights significant limitations in the current capabilities of multimodal LLMs in reading and interpreting business-related visual data. These findings have substantial practical implications:

• Human Oversight: Despite their advanced capabilities, GPT-4o and Gemini Flash-1.5 are not yet reliable enough for standalone use in high-stakes business applications. Human oversight remains essential to ensure the accuracy of data interpretation.
• Tool Development: For these models to be effectively integrated into business software, enhancements in their ability to process and accurately interpret complex and unlabeled charts are crucial.

Future Developments

Potential future developments in AI could aim at improving the precision of multimodal models through:

• Enhanced Pre-training: Further cross-modal pre-training on diverse and complex datasets might help reduce error rates.
• Specialized Modules: Developing specialized modules within these models focused exclusively on interpreting specific types of visual data could also be beneficial.
• Human-AI Collaboration: Future systems might increasingly rely on a hybrid approach, leveraging AI for initial interpretations and human intelligence for validation and correction.

Conclusion

The paper provides a meticulous analysis of current multimodal AI models' capabilities and limitations in reading business charts. Despite their advanced capabilities, GPT-4o and Gemini Flash-1.5 demonstrate substantial limitations in accuracy, underscoring the necessity of human oversight. This research outlines a clear pathway for future improvements and presents significant considerations for practical business applications requiring high data accuracy.